System and method for associating music with brain-state data

ABSTRACT

A system and method may be provided for associating bio-signal data (e.g. EEG brain scan data) from at least one user with at least one music data item (e.g. song, or piece of music). By associating bio-signal data, or emotions determined therefrom, with music, the system may establish a data store of music associated with emotions. That database may then be leveraged upon determining that a user is feeling a particular emotion through an EEG scan. When a particular emotion is detected in EEG data of a user, the system may then respond based at least partly on the same or similar emotion being associated with one or more music data items in the system. For example, the system may recommend a particular song associated with the same emotion presently being experienced by the user.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part of U.S. application Ser. No. 16/394,563, filed Apr. 25, 2019, which is a continuation of U.S. application Ser. No. 14/693,480, filed Apr. 22, 2015, which claims the benefit of U.S. Provisional Application No. 61/982,631, filed Apr. 22, 2014, each of which are incorporated herein by reference in their entirety.

FIELD

Embodiments described herein relate to bio-signal collection methods, and systems that utilize bio-signal data. Embodiments described herein relate more particularly to utilizing bio-signal data to control a computer response.

INTRODUCTION

Bio-signals are signals that are generated by biological beings that can be measured and monitored. Electroencephalographs, galvanometers, and electrocardiographs are examples of devices that are used to measure and monitor bio-signals generated by humans.

A human brain generates bio-signals such as electrical patterns, which may be measured or monitored using an electroencephalogram (EEG). These electrical patterns, or brainwaves, are measurable by devices such as an EEG. Typically, an EEG will measure brainwaves in an analog form. Then, these brainwaves may be analyzed either in their original analog form or in a digital form after an analog to digital conversion.

Measuring and analyzing bio-signals such as brainwave patterns can have a variety of practical applications. For example, brain computer interfaces (BCI) have been developed that allow users to control devices and computers using brainwave signals.

SUMMARY

According to an aspect, there is provided a computer-implemented method for sharing a user experience, comprising: receiving bio-signal data for a plurality of users from a plurality of bio-signal sensors; receiving sound data representing sound experienced by each of the users; determining a physiological state of each of the users based at least in part on the bio-signal data of that user; determining an environmental state associated with each of the users based at least in part on the sound data; using a rules engine, determining if a condition is met, based at least in part on the physiological state of each of the users and the environmental state associated with each user; upon the condition being met, executing an associated action including generating a sensory signal; and sending the sensory signal to transducers to output a sensory output to each of the users based on the sensory signal.

In some embodiments, the method further comprises: detecting an EEG response based on the bio-signal data; and correlating a time of the EEG response to the sound data to compute a segment of sound data corresponding to the EEG response, wherein the sensory signal comprises an audio signal, the audio signal based at least in part on the EEG response and the corresponding segment of sound data.

In some embodiments, the sound data is received from an acoustic sensor proximate one or more of the users.

In some embodiments, the bio-signal data includes at least one of brain metrics, heart metrics, or breath metrics.

In some embodiments, the sensory output includes at least one of light, sound, tactile pressure, tactile vibration, or scent.

In some embodiments, the method further comprises: receiving image data from one or more image sensors proximate one or more of the users, wherein the determining the environmental state associated with each of the users is based at least in part on the image data.

In some embodiments, the method further comprises: receiving image data from one or more image sensors proximate one or more of the users, wherein the physiological state associated with each of the users is based at least in part on the image data.

In some embodiments, the method further comprises: receiving tactile signals from one or more tactile transducers proximate one or more of the users, wherein the determining the physiological state associated with each of the users is based at least in part on the tactile signals.

In some embodiments, the sensory output to each of the users is individualized to each user based at least in part on a user profile of that user.

According to another aspect, there is provided a non-transitory computer-readable medium having computer executable instructions stored thereon for execution by one or more computing devices, that when executed perform a method as described herein.

According to another aspect, there is provided a shared user experience system, comprising: a plurality of bio-signal sensors to capture bio-signal data from a plurality of users; an input device for at least one user to change a state of a rules engine; a plurality of transducers to output a sensory output to each of the users; and a computing device in communication with the bio-signal sensors, the acoustic sensor, and the transducers, the computing device configured to: receive bio-signal data for the users from the bio-signal sensors; receive input data from the input device; determine a physiological state of each of the users based at least in part on the bio-signal data of that user; using the rules engine, determine if a condition is met, based at least in part on the physiological state of each of the users; and upon the condition being met, execute an associated action including generate a sensory signal and send the sensory signal to the transducers to output a sensory output to the users based on the sensory signal.

In some embodiments, the bio-signal data includes at least one of brain metrics, heart metrics, or breath metrics.

In some embodiments, the sensory output includes one or more of visual, audio, tactile, heat, or scent.

In some embodiments, the system further comprises a tactile transducer to receive tactile input from the users, wherein the computing device is in communication with the tactile transducer and is further configured to: receive tactile signals from the tactile transducer, and the physiological state of each of the users is based at least in part on the tactile signals of that user.

In some embodiments, the computing device is further configured to: determine a group physiological state of the users based at least in part on the bio-signal data of each user and the state of the rules engine, wherein the condition is based at least in part on the group physiological state and the state of the rules engine.

In some embodiments, the computing device is further configured to: receive sound data from one or more acoustic sensors and determine an environmental state associated with each of the users based at least in part on the sound data, wherein the condition is based at least in part on the physiological state of each of the users and the environmental state associated with each user.

In some embodiments, the physiological state of each of the users is based at least in part on the bio-signal data of that user and time-stamped sound data as measured by the one or more acoustic sensors representing sound delivered to each of the users, the bio-signal data time-stamped and synchronized with the time-stamped sound data.

In some embodiments, one or more of the users are located remotely from each other.

In some embodiments, the computing device is in communication with the bio-signal sensors, the acoustic sensor, and the transducers over a network.

In some embodiments, the condition is based at least in part on a performance of the physiological states of the users as compared to a target physiological state.

In accordance with an aspect of the embodiments described herein, a system is provided with a database that is built of a user's EEG response to specific musical pieces. Combined with other information such as the user's music selections, personality questions, and demographic information, a list of songs can be recommended. The songs the system recommends may be based on the current emotional state of the user and the desired state of the user. In addition users can over-ride the predictions of the system helping improve its prediction algorithms.

In accordance with an aspect of the embodiments described herein, there is provided an intelligent music system. The system may have at least one bio-signal sensor configured to capture bio-signal sensor data from at least one user. The system may have an input receiver configured to receive music data and the bio-signal sensor data, the music data and the bio-signal sensor data being temporally defined such that the music data corresponds temporally to at least a portion of the bio-signal sensor data. The system may have at least one processor configured to provide: a music processor to segment the music data into a plurality of time epochs of music, each epoch of music linked to a time stamp; a sonic feature extractor to, for each epoch of music, extract a set of sonic features; a biological feature extractor to extract, for each epoch of music, a set of biological features from the bio-signal sensor data using the time stamp for the respective epoch of music; a metadata extractor to extract metadata from the music data; a user feature extractor to extract a set of user attributes from the music data and the bio-signal sensor data, the user attributes comprising one or more user actions taken during playback of the music data; a machine learning engine to transform the set of sonic features, the set of biological features, the set of metadata, and the set of user attributes into, for each epoch of music, a set of categories that the respective epoch belongs to using one or more predictive models to predict a user reaction of music; and a music recommendation engine configured to provide at least one music recommendation based on the set of labels or classes.

In some embodiments, the input receiver may be configured to receive a target emotional state and wherein the system further comprises a music controller to interact with the music recommendation engine to provide at least one music recommendation based on the target emotional state.

In some embodiments, the music processor may be configured to identify a selection of songs from the music data and add a temporal history to the selection of songs, the temporal history indicating a date and time a user of the at least one users listened to or otherwise selected songs of the selection of songs and an order of the selection of songs, wherein the predictive models use a temporal model for the temporal history the selection of songs.

In some embodiments, the selection of songs provides at least a portion of the metadata used for the recommendation.

In some embodiments, the at least one processor may be configured to provide the one or more predictive models has a learning mode and an operational mode.

In some embodiments, each of the categories may be nested in a hierarchy of nodes or an ordered list of probabilities of the respective category.

In some embodiments, the system may have a plurality of bio-signal sensors configured to capture bio-signal sensor data from a plurality of users and correlate a portion of the bio-signal sensor data across the plurality of users, the portion being mapped to one or more epochs of music, wherein the machine learning engine transform the portion of the bio-signal sensor data across the plurality of users to a common set of categories for use in music recommendations.

In some embodiments, the system may have a music effect controller to influence user state by playback or recording of music based the at least one music recommendation.

In another aspect, embodiments described herein may provide an intelligent music system. The system may have at least one bio-signal sensor comprising at least one brainwave sensor. The system may have at least one computing device in communication with the least one bio-signal sensor to continuously receive bio-signal data comprising brainwave data of at least one user. The at least one computing device may be configured to:

define a profile for the at least one user comprising the brainwave data, and user attributes, the brainwave data linked to a timeline; detect an EEG response as a segment of the brainwave data at a time period on the timeline, the EEG response defining a change in brain state; correlate the time period to music data to compute a segment of music data corresponding to the segment of the brainwave data of the EEG response; identify a selection of music data using the segment of music data and the user attributes; and transmit signals defining a recommendation of a music data item based on the selection of music data.

In some embodiments, at least one computing device is configured to take multiple samples of the brainwave data a different times to detect a plurality of EEG responses and timestamp any detected EEG response.

In some embodiments, the user attributes may have data fields defining music selections, personality data, and demographic data.

In some embodiments, the EEG response defines a current emotional state of the user, and the selection of music data is linked to a desired emotional state relative to the current emotional state.

In some embodiments, the at least one computing device is configured to receive user feedback to reject or accept the recommendation based on the selection of music data, and refine subsequent selections of music data based on the user feedback.

In some embodiments, the at least one computing device configured to identify the selection of music data by identifying users that have similar EEG responses to the detected EEG response.

In some embodiments, the user attributes have data fields defining at least one mental state, and the selection of music data is linked to treatment for the at least one mental state.

In some embodiments, in the computing device may be configured to determine a correspondence between the received brainwave data and historical data available to the system associated with at least one second user; and trigger a user correspondence action based at least partly on the determined correspondence.

In some embodiments, the at least one computing device may be configured to provide at least one digital content item to at least one user at the at least one computing device, determine at least one emotion exhibited by the received brainwave data; and associate the at least one emotion with the at least one digital content item.

In some embodiments, the at least one bio-signal sensors has sensors for receiving data defining physiological measurements of the user.

In some embodiments, the system has cloud data storage connected to the at least one computing device, the cloud data storage storing the profile, the music data and the brainwave data.

In some embodiments, that system has an audio input device to receive audio signals corresponding to the music data.

In some embodiments, at least one computing device may be configured to generate a data structure with a tag on the music data, the tag defining an emotional state based on the EEG response.

In some embodiments, the EEG response defines a current physical state of the user and wherein the at least one computing device is configured to determine the recommendation based on a desired physical state relative to the current physical state.

In some embodiments, the system has an interface to a music platform for triggering download or purchase of the music data item of the recommendation.

In some embodiments, the system has an interface for displaying a current emotional state of the user based on the EEG response.

In another aspect, embodiments described herein may provide a system with a plurality of bio-signal sensors, each bio-signal sensor comprising at least one brainwave sensor; and at least one computing device in communication with the plurality of bio-signals sensor to continuously receive bio-signal data comprising brainwave data of a plurality of users. The at least one computing device may be configured to: detect an EEG response as a segment of the brainwave data at a time period; correlate the time period to music data to compute a segment of music data corresponding to the segment of the brainwave data of the EEG response; determine a collective emotional state of the plurality of users; and generate a music data item using the segment of music data and the collective emotional state.

In accordance with an aspect of embodiments described herein, the system of the present disclosure may find other users that have similar EEG responses to music as the user. For example, this can be added to web sites like Spotify and or dating web sites. For example, the system may only allow people into a private forum web site if they have had a strong enough emotional response to a song.

In accordance with an aspect of embodiments described herein, the system of the present disclosure may change mood through music, for example, for treating depression using music therapy.

In accordance with an aspect of embodiments described herein, the system of the present disclosure may use EEG for marketing or creation of music, by studying the EEG responses of people to new music to provide feedback to the creative process.

In accordance with an aspect of the embodiments described herein, there is provided a system with at least one computing device; at least one bio-signal sensor in communication with the at least one computing device; the at least one computing device configured to: receive bio-signal data of the at least one user from the at least one bio-signal sensor, at least one of the at least one bio-signal sensor comprising a brainwave sensor, and the received bio-signal data comprising at least brainwave data of the at least one user; receive other information from or about the at least one user; and recommend at least one selection of music data to the at least one user based at least partly on the received bio-signal data and the other information.

In accordance with an aspect of embodiments described herein, there is provided the system of the present disclosure wherein the each of plurality of music data is associated with treatment for at least one mental state; the recommending comprising recommending at least one of the plurality of music data based at least partly on a determined correspondence between the respective associated mental state to be treated and the received bio-signal data.

In accordance with an aspect of embodiments described herein, there is provided a system comprising: at least one computing device; at least one bio-signal sensor in communication with the at least one computing device; the at least one computing device configured to: receive bio-signal data of the at least one user from the at least one bio-signal sensor, at least one of the at least one bio-signal sensor comprising a brainwave sensor, and the received bio-signal data comprising at least brainwave data of the at least one user; determine a correspondence between the received bio-signal data and bio-signal data available to the system associated with at least one second user; and trigger a user correspondence action based at least partly on the determined correspondence.

In accordance with an aspect of embodiments described herein, there is provided a system comprising: at least one computing device; at least one bio-signal sensor in communication with the at least one computing device; the at least one computing device configured to: present at least one digital content item to at least one user at the at least one computing device; receive bio-signal data of the at least one user from the at least one bio-signal sensor, at least one of the at least one bio-signal sensor comprising a brainwave sensor, and the received bio-signal data comprising at least brainwave data of the at least one user; determine at least one emotion exhibited by the received bio-signal data; and associate the at least one emotion with the presented at least one digital content item.

In accordance with an aspect of embodiments described herein, there is provided a method performed by at least one computing device, the method comprising the steps of the at least one computing device from the system of the present disclosure.

In this respect, before explaining at least one embodiment of the disclosure in detail, it is to be understood that the disclosure is not limited in its application to the details of construction and to the arrangements of the components set forth in the following description or illustrated in the drawings. The disclosure is capable of other embodiments and of being practiced and carried out in various ways. Also, it is to be understood that the phraseology and terminology employed herein are for the purpose of description and should not be regarded as limiting.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments will now be described, by way of example only, with reference to the attached figures, wherein:

FIG. 1 illustrates a chart depicting valence-arousal dimensions for quantifying emotions;

FIG. 2 shows an exemplary system view for a system in accordance with embodiments described herein;

FIG. 3 shows an exemplary entry of brain state in a database in accordance with embodiments described herein;

FIG. 4 shows a system view of user override of music recommendation in accordance with embodiments described herein;

FIG. 5 shows a table indicating user emotions before listening and after listening to a particular song;

FIG. 6 illustrates an exemplary architecture in accordance with embodiments described herein;

FIG. 7 illustrates an exemplary architecture of the present in accordance with an aspect of the embodiments described herein;

FIG. 8 illustrates an exemplary architecture in accordance with embodiments described herein;

FIG. 9 illustrates a computer used to implement aspects of the embodiments described herein;

FIG. 10 shows an example of a user interface for describing the current user state and target state according to some embodiments;

FIG. 11 shows an example of a user interface with a valence arousal interactive space for user input according to some embodiments;

FIG. 12 shows a system view of extracting sonic, biological and user actions and using machine learning to determine a predictive model of the music data and the user's reaction to it;

FIG. 13 shows a time representation of the features shown in FIG. 12;

FIG. 14 shows a graph with an example of 9 different users;

FIG. 15 is a system view of a control system that changes the effects of the music to match a user's desired target state;

FIG. 16 is a high-level block diagram of an example embodiment of a shared user experience system;

FIG. 17 is a schematic diagram of an operating environment of an example embodiment;

FIG. 18 illustrates the organization of software of a computing device of FIG. 9;

FIG. 19 is a flow chart of a method for generating a shared user experience, by the software of FIG. 18; and

FIGS. 20A to 20F are examples of visualizations during a shared user experience.

In the drawings, embodiments are illustrated by way of example. It is to be expressly understood that the description and drawings are only for the purpose of illustration and as an aid to understanding, and are not intended as a definition of the limits of the invention.

DETAILED DESCRIPTION

A system and method is described associating bio-signal data (e.g. EEG brain scan data) from at least one user with at least one music data item (e.g. song, or piece of music). “Music” as used herein is understood to encompass any form of music, sound, or audio, or the like, either in a recorded format, for example, on a digital or analog medium, or a live representation. By associating bio-signal data, or emotions determined therefrom, with music, the system may establish a database of music associated with emotions. That database may then be leveraged upon determining that a user is feeling a particular emotion through an EEG scan. When a particular emotion is detected in EEG data of a user, the system may then respond based at least partly on the same or similar emotion being associated with one or more music data items in the system. For example, the system may recommend a particular song associated with the same emotion presently being experienced by the user. The system may then also begin playing that song. The database of music data and bio-signal or emotion data may be stored in a local computer or accessed on one or more servers, such as in the cloud. The music may be music that the user has access to or not. If the user does not have access to play the particular music data item recommended for playback, the system may also provide one or more options to the user for acquiring access to the recommended music data item (e.g. offer a choice to purchase the song or refer the user to a third-party service, retailer, or provider that may be able to provide access to the song to the user).

Systems and methods described herein may recommend audio or music on the basis of users physiological state (for example, bio-signal data) and the users' environment to generate a group experience, such as a sensory output, for the users, for example, for relaxation, meditation, therapy, sound healing, mental performance training, yoga, physical practice, group integration, and the like. An analysis of a user's environment includes, for example, ambient or generated sound that a user experiences.

A computer system is provided that is implemented by one or more computing devices. The computing devices may include one or more client or server computers in communication with one another over a near-field, local, wireless, wired, or wide-area computer network, such as the Internet, and at least one of the computers is configured to receive signals from sensors worn by a user. In an implementation, the sensors include one more sensors. In some embodiments, the one or more sensors include bio-signal sensors, such as electroencephalogram (EEG) sensors, galvanometer sensors, electrocardiograph sensors, heart rate sensors, eye-tracking sensors, blood pressure sensors, pedometers, gyroscopes, accelerometer, magnetometer, camera and any other type of sensor. The sensors may be of various types, including: electrical bio-signal sensor in electrical contact with the user's skin; capacitive bio-signal sensor in capacitive contact with the user's skin; blood flow sensor measuring properties of the user's blood flow; and wireless communication sensor placed sub-dermally underneath the user's skin. Other sensor types may be possible. The sensors may be connected to a wearable computing device, such as a wearable headset or headband computer worn by the user. The sensors may be connected to the headset by wires or wirelessly. The headset may further be in communication with another computing device, such as a laptop, tablet, or mobile phone such that data sensed by the headset through the sensors may be communicated to the other computing device for processing at the computing device, or at one or more computer servers, or as input to the other computing device or to another computing device. The one or more computer servers may include local, remote, cloud based or software as a service platform (SAAS) servers. Embodiments of the system may provide for the collection, analysis, and association of particular bio-signal and non-bio-signal data with specific mental states for both individual users and user groups. The collected data, analyzed data or functionality of the systems and methods may be shared with others, such as third party applications and other users. Connections between any of the computing devices, internal sensors (contained within the wearable computing device), external sensors (contained outside the wearable computing device), user effectors, and any servers may be encrypted. Collected and analyzed data may be used to build a user profile that is specific to a user. The user profile data may be analyzed, such as by machine learning algorithms, either individually or in the aggregate to function as a BCI, or to improve the algorithms used in the analysis. Optionally, the data, analyzed results, and functionality associated with the system can be shared with third party applications and other organizations through an API. One or more user effectors may also be provided at the wearable computing device or other local computing device for providing feedback to the user, for example, to vibrate or provide some audio or visual indication to assist the user in achieving a particular mental state, such as a meditative state.

The wearable computing device may include a camera, a display, and bio-signal measuring means to sample a user's environment as well as the user's bio-signals, determining the user's state and context through sensors and user input. The wearable computing device may include at least one user-facing camera to track eye movement or other physiological measures. In a particular aspect of the embodiments described herein, the wearable computing device may be in a form resembling eyeglasses wearable on the user's face. Optionally, at least one camera may be oriented to generally align with the user's field of view.

In another aspect of the embodiments described herein, the wearable computing device may be in a form of at least one sensor adapted to being placed at or adhered to the user's head or face. Each sensor may optionally communicate with one another either through wires or wirelessly. Each sensor may optionally communicate with a controller device either through wires or wirelessly. The controller device may be mounted to the wearable computing device in order to reside at or near the user's head or face. Alternatively, the controller device may be located elsewhere on the user's body, such as in a bag or pocket of the user's clothing. The controller device may also be disposed somewhere outside the user's body. For example, the sensors may monitor the user, storing data in local storage mounted to the wearable computing device, and once moving into proximity with the controller device, the sensors, or a transmitter of the wearable computing device may transmit stored data to the controller device for processing. In this implementation, the wearable computing device would be predominantly usable by the user when located nearby the controller device.

The wearable computing device may include a camera, a display and bio-signal measuring means. At least one of the bio-signal measuring means may employ at least one sensor in order to measure brain activity. Brain activity may be measured through electroencephalography (“EEG”) techniques electrically, or through functional near-infrared spectroscopy (“fNIR”) techniques measuring relative changes in hemoglobin concentration through the use of near infrared light attenuation. A sensor employing pulse oximetry techniques may also be employed in the wearable computing device. Optionally, the wearable computing device may include at least one sensor measuring eye activity using electrooculography (“EOG”) techniques. Other sensors tracking other types of eye movement may also be employed.

In various implementations, the wearable computing device may include a variety of other sensors and input means. For example, the wearable computing device may comprise at least one audio transducer such as a single microphone, a microphone array, a speaker, and headphones. The wearable computing device may comprise at least one inertial sensor for measuring movement of the wearable computing device. The wearable computing device may comprise at least one touch sensor for receiving touch input from the user.

The wearable computing device may sample from both the user's environment and bio-signals simultaneously or generally contemporaneously to produce sampled data. The sampled data may be analyzed by the wearable computing device in real-time or at a future predetermined time when not being worn by the user.

In addition to being worn on the user's head, the wearable computing device may be worn in a pouch or pocket or secured to the user with a strap, or held in the user's hands.

The wearable computing device may comprise user input detection methods that are adaptive and improve with use over time. Where the user attempts to command the wearable computing device, and the wearable computing device responds in an unexpected way, the user may attempt to correct the previous input by indicating that the wearable computing device response was incorrect, and retrying the initial command again. Over time, the wearable computing device may refine its understanding of particular user inputs that are corrected. Some user inputs may be easier to successfully measure with a high degree of accuracy than others. It may be preferable to assign a high-accuracy input to command the wearable computing device that the previous input was incorrect. For example, tapping the wearable computing device in a particular spot may indicate that the previous input response was incorrect. Explicit training such as with voice recognition may also be used to configure and command the wearable computing device.

In one implementation, the wearable computing device may be in a glasses-like form factor. Glasses, with or without eyeglass elements, may be well-suited on which to mount sensors as glasses may be easily mounted to the user's face, and are easily removed. Glasses may also be relatively stable in position with respect to the user's head when resting on parts of the user's nose and ears. In order to further reduce movement of the glasses, arm-portions of the glasses may grip sides or rear portions of the user's head. Resilient arm-portions may be particularly useful for achieving a suitable gripping strength, thereby minimizing movement of the glasses and any sensors mounted thereupon.

Optionally, the wearable computing device may itself only provide bio-signal sensors and a processor for processing measurements from the sensors. The wearable computing device may communicate these measurements or data derived from processing the measurements to one or more secondary devices, such as a Google Glass-style device. In any of the implementations, embodiments, or applications discussed herein, it should be understood that some actions may be carried out by a plurality of interconnected devices, or just one of the wearable computing devices of the embodiments described herein. For example, the wearable computing device may not include a display. In such an example, the wearable computing device may communicate visual information to the user through the use of a second device, such as a Google Glass-style device, which does include a display.

Sensors usable with the wearable computing device may come in various shapes and be made of various materials. For example, the sensors may be made of a conductive material, including a conductive composite like rubber or conductive metal. The sensors may also be made of metal plated or coated materials such as stainless steel, silver-silver chloride, and other materials.

In addition to or instead of processing bio-signal measurements on the wearable computing device, the wearable computing device may communicate with one or more computing devices in order to distribute, enhance, or offload the processing of the bio-signal measurements taken or received by the wearable computing device. In particular, the one or more computing devices may maintain or have access to one or more databases maintaining bio-signal processing data, instructions, algorithms, associations, or any other information which may be used or leveraged in the processing of the bio-signal measurements obtained by the wearable computing device. The computing devices may include one or more client or server computers in communication with one another over a near-field, local, wireless, wired, or wide-area computer network, such as the Internet, and at least one of the computers may be configured to receive signals from sensors of the wearable computing device.

The wearable computing device may further be in communication with another computing device, such as a laptop, tablet, or mobile phone such that data sensed by the headset through the sensors may be communicated to the other computing device for processing at the computing device, or at one or more computer servers, or as input to the other computing device or to another computing device. The one or more computer servers may include local, remote, cloud based or software as a service platform (SAAS) servers. Embodiments of the system may provide for the collection, analysis, and association of particular bio-signal and non-bio-signal data with specific mental states for both individual users and user groups. The collected data, analyzed data or functionality of the systems and methods may be shared with others, such as third party applications and other users. Connections between any of the computing devices, internal sensors (contained within the wearable computing device), external sensors (contained outside the wearable computing device), user effectors (components used to trigger a user response), and any servers may be encrypted. Collected and analyzed data may be used to build a user profile that is specific to a user. The user profile data may be analyzed, such as by machine learning algorithms, either individually or in the aggregate to function as a BCI, or to improve the algorithms used in the analysis. Optionally, the data, analyzed results, and functionality associated with the system can be shared with third party applications and other organizations through an API. One or more user effectors may also be provided at the wearable computing device or other local computing device for providing feedback to the user, for example, to vibrate or provide some audio or visual indication to assist the user in achieving a particular mental state, such as a meditative state.

A cloud-based implementation for processing and analyzing the sensor data may provide one or more advantages including: openness, flexibility, and extendibility; manageable centrally; reliability; scalability; being optimized for computing resources; having an ability to aggregate information across a number of users; and ability to connect across a number of users and find matching sub-groups of interest. While embodiments and implementations described herein may be discussed in particular non-limiting examples with respect to use of the cloud to implement aspects of the system platform, a local server, a single remote server, a SAAS platform, or any other computing device may be used instead of the cloud.

In one implementation of the system of the embodiments described herein, a Multi-modal EEG Data-Collection and Adaptive Signal Processing System (MED-CASP System) for enabling single or multi-user mobile brainwave applications may be provided for enabling BCI applications. This system platform may be implemented as a hardware and software solution that is comprised of an EEG headset such as the wearable computing device of the embodiments described herein, a client side application and a cloud service component. The client side application may be operating on a mobile or desktop computing device. The system may provide for: estimation of hemispheric asymmetries and thus facilitate measurements of emotional valence (e.g. positive vs. negative emotions); and better signal-t-noise ratio (SNR) for global measurements and thus improved access to high-beta and gamma bands, which may be particularly important for analyzing cognitive tasks such as memory, learning, and perception. It has also been found that gamma bands are an important neural correlate of mediation expertise.

In the same or another non-limiting exemplary implementation, possible MED-CASP system features may include: uploading brainwaves and associated sensor and application state data to the cloud from mobile application; downloading brainwave & associated data from the cloud; real-time brain-state classification to enable BCI in games or other applications; transmitting real-time brain-state data to other users when playing a game to enable multi-user games; sharing brainwave data with other users to enable asynchronous comparisons of results; sharing brainwave data to other organizations or third party applications and systems; and support of cloud based user profiles for storing personal information, settings and pipeline parameters that have been tuned to optimize a specific user's experience. In this way, usage of the system platform can be device independent.

Each time analysis or processing of user bio-signal data (such as brainwave data) is performed, an instance of aspects of the software implementing the analysis functionality of the embodiments described herein may be generated by the wearable computing device, initiated at either the device or the cloud, in order to analyze the user's private bio-signal data using particular analysis or processing parameters applied during the analysis or processing. For simplicity, such an instance may be referred to as an algorithm “pipeline”. Each instance of the pipeline may have an associated pipeline identifier (“ID”). Each pipeline may be associated with a particular activity type, user, bio-signal type of a particular user, application, or any other system platform-related data. Each pipeline may maintain particular pipeline parameters determined to analyze the user's bio-signal data in a particular way, consistent either with previous analysis of the particular user's bio-signal data, consistent with previous analysis of one or more other user's bio-signal data, or consistent with updated data at the cloud server derived from new or updated scientific research pertaining to the analysis of bio-signal data. Pipelines and/or pipeline parameters may be saved for future use at the client computing device or at the cloud. When a new pipeline is created for the user, the wearable computing device or the cloud may provide a new algorithm pipeline ID to be associated with the new pipeline at the cloud and at the device.

Each person's brainwaves are different, therefore requiring slightly different tunings for each user. Each person's brain may also learn over time, requiring the system platform to change algorithm parameters over time in order to continue to analyze the person's brainwaves. New parameters may be calculated based on collected data, and may form part of a user's dynamic profile (which may be called bio-signal interaction profile). This profile may be stored in the cloud, allowing each user to maintain a single profile across multiple computing devices. Other features of the same or another non-limiting exemplary implementation may include: improving algorithms through machine learning applied to collected data either on-board the client device or on the server; saving EEG data along with application state to allow a machine learning algorithm to optimize the methods that transform the user's brainwaves into usable control signals; sharing brainwave data with other applications on mobile device through a cloud services web interface; sharing brainwave data with other applications running on client devices or other devices in the trusted network to provide for the user's brainwave data to control or effect other devices; integration of data from other devices and synchronization of events with brainwave data aid in context aware analysis as well as storage and future analysis; performing time locked stimulation and analysis to support stimulus entrainment event-related potential (“ERP”) analysis; and data prioritization that maximizes the amount of useful information obtainable from an incomplete data download (i.e. data is transmitted in order of information salience). The core functionality of the MED-CASP system may be wrapped as an externally-usable library and API so that another developer may use the platform's features in the developer's application(s). The library may be a static library and API for Unity3D, iOS, Android, OSX, Windows, or any other operating system platform. The system platform may also be configured to use a pre-compiled algorithm supplied by a third party within the library, including the ability for a third party developer using the library, to use the developer's own algorithms with the library. The system platform may also support headsets from a variety of vendors; personal data security through encryption; and sharing of un-curated data (optionally using time-limited and fidelity limited access) though the sharing of encryption keys.

Optionally, the system of the present disclosure may be used to implement aspects of the systems and methods described in PCT Patent Application No. PCT/CA2013/000785, filed Sep. 16, 2013, the entirety of which is incorporated by reference herein. Accordingly, the system of the present disclosure may be or may be used with a computer network implemented system for improving the operation of one or more biofeedback computer systems. The system may include an intelligent bio-signal processing system that is operable to: capture bio-signal data and in addition optionally non-bio-signal data; and analyze the bio-signal data and non-bio-signal data, if any, so as to: extract one or more features related to at least one individual interacting with the biofeedback computer system; classify the individual based on the features by establishing one or more brain wave interaction profiles for the individual for improving the interaction of the individual with the one or more biofeedback computer systems, and initiate the storage of the brain waive interaction profiles to a database; and access one or more machine learning components or processes for further improving the interaction of the individual with the one or more biofeedback computer systems by updating automatically the brain wave interaction profiles based on detecting one or more defined interactions between the individual and the one or more of the biofeedback computer systems.

Optionally, the system of the present disclosure may be used to implement aspects of the systems and methods described in PCT Patent Application No. PCT/CA2013/001009, filed Dec. 4, 2013, the entirety of which is incorporated by reference herein. Accordingly, the system of the present disclosure may be or may be used with a computer system or method for modulating content based on a person's brainwave data, obtained by the sensors of the wearable apparatus of the present disclosure, including modifying presentation of digital content at at least one computing device. The content may also be modulated based on a set of rules maintained by or accessible to the computer system. The content may also be modulated based on user input, including through receipt of a presentation control command that may be processed by the computer system of the present disclosure to modify presentation of content. Content may also be shared with associated brain state information.

Shared User Experience

In accordance with an aspect of the embodiments described herein, and having reference to FIG. 16, there is an example embodiment of a system 100 for generating a shared user experience.

As illustrated in FIG. 16, system 100 may include a wearable device such as a head-mountable or head-worn device including at least one bio-signal sensor 102 associated with a user 101. System 100 also includes a computing device 120 in communication with bio-signal sensors 102 and transducers 104 associated with a user 101 to receive bio-signal data from bio-signal sensors 102. Computing device 120 is in communication with a remote computing device 140 and a data store 150 by way of a network 160. Network 160 may, for example, be a packet-switched network, in the form of a LAN, a WAN, the public Internet, a Virtual Private Network (VPN) or the like. One or more acoustic sensors 106, for detecting sound in an environment, and one or more image sensors 108, for example a camera or the like for capturing image data of an environment, may also be in communication with computing devices 120 and/or remote computing device 140 by way of network 160.

In some embodiments, at least one movement sensor(s) (not shown) may also be associated each user 101, for example included in a wearable device and also in communication with a computing device 120. Such movement sensors may include, for example, an accelerometer, a gyroscope, a magnetometer, an image sensor such as a camera. A camera may be a user-facing camera, for example, to track eye movement, or may be oriented to generally align with the user's field of view.

The system 100 also includes a computing device 120 in communication with bio-signal sensors 102 and transducers 104 associated with a user 101 to receive bio-signal data bio-signal sensors 102. Computing device 120 may be connected to a remote computing device 140 and a data store 150 by way of a network 160.

In some embodiments, one or more of users 101 are geographically or physically located local or proximate to each other, for example, in the same room.

Local users 101, and one or more components of system 100 described herein, may be connected and communicate by a local network.

In some embodiments, one or more of the users 101 are located remotely from each other.

As described in further detail below, system 100 may facilitate a remote experience for a user 101 over network 140 such as the internet using a mobile device, such as computing device 120, computer or smart TV, with a bio-signal and/or body and/or brain sensing device, such as bio-signal sensors 102. Sound can be sensed, mixed and processed using techniques described herein, and may help a remote user 101 feel like they are present with a group of other users 101.

Bio-signal sensors 102 capture bio-signal data (such as brainwave data) from one or more users 101 and transmit bio-signal data to computing device 120.

Bio-signal sensors 102 may be a bio-signal sensor as described herein, for example, an electrophysiological sensor of various types, including: electrical bio-signal sensor in electrical contact with the skin of user 101; capacitive bio-signal sensor in capacitive contact with the user's skin; blood flow sensor measuring properties of the user's blood flow.

For example, bio-signal sensors 102 may include one or more of electroencephalogram (EEG) sensors, galvanometer sensors, electrocardiograph sensors, heart rate sensors, eye-tracking sensors, blood pressure sensors, pedometers, gyroscopes, accelerometer, magnetometer, and any other type of suitable sensor.

Bio-signal data may include, for example, brain metrics, heart metrics, or breath metrics, such as EEG, ECG, or breath rate.

In some embodiments, bio-signals may be measured with an image sensor, such as a camera. Thus, bio-signal sensor 102 may include an image sensor.

One or more user transducers 104 may also be provided proximate a user 101 and in communication with a computing device 120 for providing feedback to the user, for example, to vibrate or provide some audio or visual indication to the user. Each user 101 may be associated with a unique one of transducers 104.

For example, transducer 104 may include at least one audio transducer or sound generator such as a single microphone, a microphone array, a speaker, and headphones, to provide auditory feedback to a user.

Transducer 104 may also generate a variety of colour and brightness of light.

In some embodiments, transducer 104 may provide a sensory output to each of the users 101 based on a sensory signal generated by computing device 120 and/or remote computing device 140, as described in further detail below.

In some embodiments, transducer 104 may be associated and proximate with one or more users 101 and may operate as a music output device for a group or users 101.

In some embodiments, transducer 104 is a music output device per individual user 101

In some embodiments, transducer 104 may be in a more physical form of a light, sound and vibration emitting orb.

In some embodiments, transducer 104 is embodied in computing device 120 (e.g., a mobile phone associated with a user 101).

In some embodiments, transducer 104 may be integral with a head-mountable device to be worn by a user 101 such as a head-mountable device suitable for also housing bio-signal sensors 102.

In some embodiments, transducer 104 may provide a visual component, and thus visual components may be added to a local group experience of users 101, or one or more remote users 101, or both.

Visual components may include video projection, lighting, live camera feed, and may be computer generated (locally or remotely). In an example, visual components may include a video feed from a group experience environment which may be streamed over network 160, such as the internet, to a remote client device, such as computing device 120.

In some embodiments, a group experience has a visual experience that is controlled by a rules engine, as described in further detail below. A similar rules engine may run on a remote client device, such as computing device 120, allowing for a local rendering that is same, similar, or different, in an example, depending on the remote client device.

Transducer 104 may be used locally by users 101 or by remote clients to provide additional stimulus and or measurement. In an example, a device such as transducer 104 may emit any of: light, sound, vibration, tactile, heat, cooling, scent.

Transducer 104 may measure and provide input data to system 100 any of movement, pressure, heart, light, sound, vibration, ECG, PPG, O2 concentration, GSR, and the like.

Transducer 104 can be output devices for music, as well as for input into the rules engines (individuals may have rules engine that operates locally as well as take input from an aggregate rules engine).

Sound producing transducers 104 in a local group experience environment may allow local users 101 to hear music or see or experience the presence of the extended group through the network or remote users 101. Remote users 101 may hear and or see or experience the presence of the group that they are connected to over network 160. Users in the extended network may have microphones (not shown) that allow users in a local group to hear the sound from the remote users, sounds could be from their ambient environments, or proximal sounds such as the sound of their breath or heartbeat. Sound data from remote user microphones may be processed for noise cancellation. Noise cancellation may be modified by data of the user's physiological state and/or the state of a rules engine.

An individual user's 101 transducer 104 may produce a stimulus that is different and/or individual to that user 101.

In some embodiments, transducer 104 may include a tactile pressure transducer such as: a cushion that you sit on; a transducer that you hold in your hands; pressure transducers in a yoga mat; strain/stretch/tension measurement in a belt, band or other exercise tool.

A transducer 104 may be implemented in a yoga block, or a tool that a user 101 may hold in the user's hand, such as a massage implement. A massage tool, for example, may incorporate both sensory and affectory transducers.

In some embodiments, transducer 104 may include a controllable atomizer or heat based vaporizer as an output transducer, for example, a rules engine as described herein controls a sage burner.

System 100 may also include acoustic sensor 106, which can be any suitable sound or audio capture component, for example a suitable microphone or series of microphones.

Acoustic sensor 106 may be a microphone or another data input to get information from a live musician or other source of audio that one or more users 101, as participants in a group experience, can hear. Acoustic sensor 106 may also be used with voice recognition to provide the rules engine with environmental information related to a condition being met.

One or more users 101 may hear sound from a sound system of speaker or other transducers, including musical instruments that are played by a person or computer actuated.

Users 101 may each have their own sound producing transducer or speaker or headphones, in combination with the other transducers, speakers, instruments, or alone (such as the case for a remote user 101).

In some embodiments, an acoustic sensor 106 such as a microphone may be used for noise cancellation, or to receive information related to an environmental context that user 101 is present in.

Sound or audio captured by acoustic sensor 106 may be recorded and stored or transmitted by computing device 120.

Image sensor 108 can be any suitable image capture component, for example a camera including CMOS or CCD sensors. Image sensor 108 may capture an image or video, such as an image of an environment in which one or more users 101 are present. In some embodiments, image sensor 108 may capture a video recording of a group experience of one or more users 101. In some embodiments, image sensor 108 is used to measure the physiological state of one or more users 101.

In some embodiments, image sensor 108 may be integrated with a computing device 120, or image sensor 108 may be implemented as a separate hardware and/or software device.

In some embodiments, system 100 may include more than one acoustic sensor 106 and image sensors 108, depending on where users 101 are located. For example, if users 101 are physically located at one or more locations, which may be remote from each other, sensors may be located proximate to users at each of one or more locations.

Computing device 120 can be any suitable electronic devices that interfaces with one another, network 160, and/or components such as bio-signal sensors 102 and transducers 104, to provide complementary functions as described herein. Computing device 120 may be a mobile computing device. For clarity in the discussion herein, mobile computing devices are commonly referred to as “mobile devices” or “devices” for brevity.

Example mobile devices include, without limitation, cellular phones, cellular smart-phones, wireless organizers, pagers, personal digital assistants, computers, laptops, handheld wireless communication devices, wirelessly enabled notebook computers, portable gaming devices, tablet computers, or any other portable electronic device with processing and communication capabilities. In at least some embodiments, mobile devices as referred to herein can also include, without limitation, peripheral devices such as displays, printers, touchscreens, projectors, digital watches, cameras, digital scanners and other types of auxiliary devices that may communicate with another computing device.

In one example, computing device 120 may be a smartphone, or one may be a smartphone and the other a peripheral device (e.g., a speaker, a keyboard, a display screen, a camera). As will be apparent, other types of computing devices 120 can be envisaged that have data processing capabilities.

In some embodiments, computing device 120 is included on a head-worn device by user 101. In other embodiments, computing device 120 is discrete from a head-worn device. When computing device 120 is discrete from the head-worn device, the computing device 120 is in wired or wireless connection with bio-signal sensors 102 of the head-worn device. When computing device 120 connects to the head-worn device wirelessly, computing device 120 includes a network controller to control communication components such as a receiver to receive bio-signal data from bio-signal sensors 102, and optionally a transmitter.

Each user 101 may be associated with a computing device of some kind, such as computing device 120.

In an example of computing device 120 embodied in a head-worn device by user 101, bio-signal signal processing may be performed by a remote computing device such as remote computing device 140, which sends to user 101, by way of computing device 120, sound information or other sensory stimulus info, such as a sensory signal as described in further detail below, and thus determines what a user 101 should hear, in an example, more wind audio or less wind audio, on the basis of what is happening with other users 101.

Each computing device 120 may be connected over a network such as network 160. A computing device 120 may be local to each user 101, for example a mobile device associated with a particular user 101, or a computing device 120 may be local to multiple users 101.

Remote computing device 140 may be a computing device, for example, a cloud computing device, connected by way of network 160 to users 101 to perform any of the functionality described herein.

Systems and methods as described herein may be performed by either or a combination of local computing device, such as computing device 120, and a remote computing device such as remote computing device 140.

In some embodiments, computing device 120 and/or remote computing device 140 may be embodied as a computing device 500, as described herein and shown in FIG. 9.

FIG. 9 is a high-level block diagram of computing device 500, for example, a mobile computing device. As will become apparent, computing device 500, under software control, may receive bio-signal data from bio-signal sensors 102 for processing by one or more processors 210 to analyze the sensor data. Processed sensor data, for example, at least one bio-signal data, may be used for generating a shared user experience.

FIG. 9 shows a computing device 500 that may include processor(s) 520, including a central processing unit (“CPU”) 502 and a graphics processing unit (GPU) 522, connected to a storage unit 504 and to a random access memory 506. The CPU 502 may process an operating system 501, application program 503, and data 523. The operating system 501, application program 503, and data 523 may be stored in storage unit 504 and loaded into memory 506, as may be required.

Computing device 500 may further include a graphics processing unit (GPU) 522 which is operatively connected to CPU 502 and to memory 506 to offload intensive image processing calculations from CPU 502 and run these calculations in parallel with CPU 502.

An operator 507, such as user 101, may interact with the computing device 500 using a video display 508 connected by a video interface 505, and various input/output devices such as a keyboard 510, mouse 512, and disk drive or solid state drive 514 connected by an I/O interface 509. Mouse 512 may be configured to control movement of a cursor in the video display 508, and to operate various graphical user interface (GUI) controls appearing in the video display 508 with a mouse button. The disk drive or solid state drive 514 may be configured to accept computer readable media 516. Computing device 500 may form part of a network via a network interface 511, allowing computing device 500 to communicate with other suitably configured data processing systems.

FIG. 18 depicts a simplified organization of example software components and data stored within memory 506 of computing device 500. As illustrated, these software components may include operating system (OS) software 310, signal processing software 320, rules engine 330, sensory signal generation software 340.

Operating system 310 may allow basic communication and application operations related to the computing device. Generally, operating system 310 is responsible for determining the functions and features available at device 500, such as keyboards, touch screen, synchronization with applications, email, text messaging and other communication features as will be envisaged by a person skilled in the art. In an embodiment, operating system 310 may be Android™ operating system software, Linux operating system software, BSD derivative operating system software, iOS™ operating system software, or any other suitable operating system software. In embodiments in which an Android operating system platform is in use, software components described herein may be implemented using features of a framework API (Application Programming Interface) for the Android platform.

Signal processing software 320 may be configured to adapt processor(s) 520 to receive sensor data from bio-signal sensors 102 and environmental data from acoustic sensor 106 and/or image sensor 108 for processing.

Computing device 500 may receive bio-signal data for users 101, for example, from bio-signal sensors 102 to determine a physiological state of each user 101 based at least in part on the bio-signal data of that user 101.

The physiological state of a user 101 may be based at least in part on changes in an electrical potential of the bio-signal data, and the changes in the electrical potential of the bio-signal data are an event-related potential as a result of a sensory, cognitive or motor event.

The physiological state of a user 101 may be based at least in part on image data received from image sensors 108 proximate one or more of users 101.

The physiological state of a user 101 may be based at least in part on tactile signals received from tactile transducers proximate one or more users 101.

The physiological state of each of the users 101 may be determined based at least in part on the bio-signal data of that user and the sound that the user is experiencing as measured by the one or more acoustic sensors 106. The bio-signal data may be time-stamped and synchronized with time-stamped sound data.

Computing device 500 may also determine a physiological state of a group of users 101, based at least part on bio-signal data of each user 101 and a state of rules engine 330.

Computing device 500 may receive sound data, for example, from acoustic sensor 106 that detects sound proximate one or more users 101 representing sound delivered to or experienced by each of the users 101. In some embodiments, sound data may represent sound delivered by computing device 500 to one or more users 101, such as by way of a sound generator, for example a headphone earpiece or a speaker.

Sound data may be captured from acoustic sensor 106 and/or from the sound generating aspect of the system (sound that would come out of headphones or other audio transducer, for example).

In some embodiments, computing device 500 embodied as remote computing device 140 receives sound data from computing device 120 local one or more users 101.

Computing device 500 may also receive image data, for example, from image sensor 108, to determine an environmental state associated with each of the users 101 based at least in part on the sound data and the image data.

In some embodiments, the environmental state may be determined based on light data received from one or more light sensors proximate one or more users 101.

In an example, bio-signal data such as brainwave data received from bio-signal sensors 102 for more than one user 101 is aggregated.

Rules engine 330 may be configured to determine if a condition is met, and upon the condition being met, executing an associated action or workflow.

A client-side rules engine may be provided in, for example, a mobile application at computing device 120. A cloud-side rules engine may be provided, for example, in a remote computing device 140.

In some embodiments, rules engine 330 determines if a condition is met, for example, based at least in part on a physiological state of each of users 101 and an environmental state associated with each user 101.

Upon a condition being met, a workflow is applied, or an associated action is executed, in an example, generating a sensory signal.

As such, outputs are triggered by specific events. Rules engine 330 may operate as a pattern detector to look for particular events to trigger actions. In some embodiments, ultimately what is generated by sensory signal generation software 340 is a sound and light signal.

A rules engine 330 may be embodied as computer hardware and/or computer software. Rules engine 330 may: manage profiles of users 101; determine what data is to be transmitted or saved; determine which devices, such as computing device 120 and remote computing device, or users 101 any bio-signal or non-bio-signal data may be provided to; determine what results are presented to a user 101; control the data that is provided for analysis; control what sensors, such as bio-signal sensors 102, to use; determine a learning goal; and determine a group target.

Rules engine 330 may be configured to receive information regarding what sensors or other resources are available.

Rules engine 330 may provide rules that use the context of the user (i.e. emotional state, global positioning system (“GPS”) coordinates, time of day, open apps on the device, prior goals that the user has stated) to determine how to analyze the sensor data (i.e. which algorithm pipeline to apply to which sensor). Rules engine 330 may also have rules to determine which results are displayed and or stored locally on computing device 120 and which are stored in a server or remote computing device 140.

Rules engine 330 inputs may include: physiological data and environmental data on one or more users 101, from signal processing software 320; local user 101 profile; and local app information (such as status (ON or OFF) identifier, parameters).

Rules engine 330 outputs may include: which user data to send to user actuators; which data to save locally; which data to transmit to the cloud; and actions related to generation of a sensory signal by sensory signal generation software 340.

There may be an interface from the rules engine 330 to each application operating on computing device 120. Rules engine 330 may be an app that runs across all other apps that helps the user reach some goals.

Rules engine 330 may operate in a number of different modes, which may be dependent on an experience or exercise for one or more of users 101.

In an example, modes of operation of rules engine 330 might have timeline associated with them. Rules engine 330 will make decision of state of the users 101 and what time it is.

In another example, a mode of operation may be dependent on a group or network affect, for example, based at least in part on how close a group of users 101 is to a particular group state.

In some embodiments, an input device may be provided for at least one user 101 to change a state of rules engine 330.

In some embodiments, audio sensors may be connected to a voice recognition system such that key words, phrases or meanings may affect the state of rules engine 330 (or in part).

Rules engine 330 may determine if a condition is met based on a group physiological state of users 101 and the state of rules engine 330.

Upon a condition being met, a sensory signal may be generated by way of sensory signal generation software 340, based on the state of rules engine 330 and send output to users 101, for example, by way of transducers 104.

Upon a condition being met, a sensory signal may be generated that affects the environment of the group of users 101. Remote users 101 may render such a sensory signal differently than local users 101, depending on their setup.

Upon a condition being met, a sensory signal may be generated based on the state of rules engine 330 and the state of an individual user 101, and a customised output sent to each of the users 101.

In some embodiments, rules engine 330 may determine a condition based on location(s) of users 101. For example, if all users 101 are local to each other, for example, physically co-located, and in the same sensor environment.

In another example, if one or more users 101 are not physically co-located, more data may be collected regarding each user's environment, for example, by way of acoustic sensor 106 and/or image sensor 108, to send to one or more other users 101 to recreate the environment with more data, which may recreate the environment better.

In some embodiments, a condition of rules engine 330 is based at least in part on a performance of the physiological states of the users 101 as compared to a target physiological state.

In a broad aspect, rules engine 330 may generate an indication of how other users 101 are performing, for example, in the context of a group meditation.

Rules engine 330 may generate an indication of a physiological state and/or environmental state of one user 101 to another user 101 to share data between users 101. In this way, a user 101 may “hear” another user's 101 meditation. For example, rules engine 330 may start and end a meditation session for multiple users 101, and transmit shared data to users 101.

In an example, shared data may include bio-signal data representing brainwaves or brain state of a user 101. In some embodiments, shared data may include data such as a wind sound state, rain sound state, or a bird sound state associated with a user 101. In some embodiments, each user 101 may be mapped to a unique sound identifier.

By routing a signal between users 101, rules engine 330 may allow for a user's experience to be “rendered” locally for another user 101.

In some embodiments, rules engine 330 may be used help user's 101 converge to synchronized breathing, or better heart/brain coherence based on a target that rules engine 330 is trying to drive group towards, in an example, using a feedback loop.

In some embodiments, each user 101 gets sound or music feedback on their physiological state from an audio transducer, such as transducer 104, with a condition being met is the audio environment is too loud, and the associated action is to reduce the volume of the audio that is being sent to the audio output transducer.

Rules engine 330 may be configured to provide a music or audio selection for a user 101. In some embodiments, audio selection may include audio synthesis and/or adaptation to generate audio or a musical score.

In a non-limiting exemplary implementation, rules engine 330 may be used in an audio environment manager. The rules engine 330 may assist in choosing music based on the user's mood, or determine a user goal based on what is the person doing (i.e. exercising, working). If exercising the goal may be to boost energy. If working the goal may be to cultivate a state of stable high attention.

In some embodiments, a network of bio-signal data from users 100 is aggregated over a network. Aggregate data may be used to affect an output, for example, music that is played, using rules engine 330. Rules engine 330 may function as an algorithm that supports a therapeutic function or training or experiential goal. This music algorithm may change or some way at affect the music track depending on the aggregate of bio-signal data, determined physiological data, environmental data (such as sound in an environment) and possibly with the input of a human guide or instructor providing external input, for example, by way of an external computing device (not shown).

In some embodiments, rules engine 330 may generate a music recommendation for a user 101 using any of the music recommendation techniques, as described herein, for example, by detecting an emotion state of a user 101, and making an appropriate music recommendation on the basis of the user's responses.

A user 101 may have different desirable mental states that she wants to achieve depending on their context (i.e. condition). In some embodiments, a desirable mental state is a group experience shared amongst more than one user 101. When a condition is true, the workflow associated with the condition is executed.

A rules engine 330 output, for example, a music recommendation, may be made on the basis of a user's emotional response, determined as described herein.

In some embodiments, rules engine 330 may further be configured to implement a meditation competition or game, either collaborative or competitive, using rules to rebalance teams during a group meditation session. For example, rules engine 330 may implement a meditation score handicap for a particular user based on their performance.

In some embodiments, an action executed by rules engine 330 specifies a machine learning approach, and whether supervised learning or unsupervised learning is to be applied.

In an example, if supervised learning is executed, the workflow also specifies the method used to label sections of biological signals and extract features, the metric to determine performance and any measures or signals not to consider in the machine learning workflow.

Sensory signal generation software 340 may be configured to generate a sensory signal, for example, to cause an audio, visual, or tactile output for a user 101.

A sensory signal may be generated based at least in part on an output of rules engine 330. For example, selection of an audio signal to transmit to a user 101 on the basis of a music recommendation from rules engine 330.

In some embodiments, sensory signal generation software 340 generates an output signal to computing device 120 for transducer 104 to provide a sensory output to user 101, which may be individual and/or unique to that user 101 based at least in part on physiological data of a group of users 101 and the state of an environment in which user 101 is present.

In some embodiments, sensory output to users 101 is individualized to each user 101 based at least in part on the physiological state of that user 101.

In some embodiments, data store 150 is connected to computing device 120 and/or remote computing device 140 by way of network 160.

In some embodiments, data store 150, sometimes called working memory, maintains one or more sets of rules about behaviour, data about current state or knowledge, and/or a rule interpreter. A ruler interpreter may provide a mechanism for prioritizing productions when more than one is triggered.

Data store 150 may further store bio-signal data, physiological data, and/or environmental data related to users 101 or other users.

FIG. 17 illustrates an example operating environment of an embodiment of system 100, in use.

Dotted lines in FIG. 17 represent signal or data connectivity either wired/wireless, local or internet, for example, network 160.

One or more remote computing devices 140 may include one or more cloud computers or data services, if needed for the application.

FIG. 17 illustrates computing devices 120 associated with users, labeled as U0-U6, having local computer or computers for a particular user used to support user interface and local processing and connectivity to create a shared user experience. A user 101 may have at least a mobile phone and a computer (such as computing device 120)+biosensor (such as bio-signal sensors 102) integrated device.

In some embodiments, group setup such as depicted for U0-U4 may have one local network computer, such as computing device 120, for signal aggregation, processing, and experience creation, supporting the rules engine and, bio-signal processing and machine learning, music selection, synthesis and adaptation. Integrated bio-signal +computer devices connect wirelessly to the local computer system as transducers 104, such as individual sound producing and light producing feedback elements. Transducers 104 such as surround sound speaker in the room support the creation of the internal sound field that all users 101, as participants, experience as well as provide a sonic connection for users 101 that are not present in the room (U5-U8).

Device M1 is an example of a combined acoustic sensor 106 and image sensor 108, for example, a combined microphone+video capture device, providing sound and image information from inside the room to support the experience algorithms and the data required to create a realistic shared audio/visual experience for the remote users 101, as well as record the experience such that it can be re-experience in a non-live scenario.

U0 and U1 are computing devices 120 associated with a musician and vocal guide respectively, working with the rules engine to create a guided interactive musical experience that helps people engage in a meditation practice. U0's instrument is audible directly to the participant in room as well as through the room's sound system. They are audible to the remote user via M1 as well as microphone and or sensors on the instrument. U1 has a vocal microphone as well as an independent user interface to control and interact with rules engine and or observe data regarding the system and or users participating. U0 and U1 may or may not have a bio-signal sensing devices, such as bio-signal sensor 102.

FIG. 17 also illustrates transducer(s) 104 illustrated as curved lines representing visual light emitting devices, for example, projected light displays, or other suitable apparatus.

User 101 associated with U5 is taking part in the experience as a remote participant, using their smart TV, and an integrated computing device 120 computer and bio-signal sensor 102 bio-signal device.

User 101 associated with U6 is taking part in the experience as a remote participant using their smartphone, computing device 120, a wireless computer integrated brain sensing device including bio-signal sensors 102, and transducers 104 including a wireless computer integrated heart sensing wrist worn device, and wireless headphones.

User 101 associated with U7 is taking part in their bed that has integrated vibrational transducers 104 to stimulate the body in congruence with the auditory experience. Their hands are being stimulated by a combined vibrational transducer and auditory speaker. They have an integrated biosignal+computer brain and heart sensing headband that interfaces with their local computing device 120, connected to the experience network 160.

User 101 associated with U8 is taking part in the experience as a remote participant using a VR headset combined with computer and biosignal device, and data made available through M1 (spatial audio and 360 camera view).

System 100 may be used to create a shared user experience, for example, a group experience for relaxation, mediation, therapy, sound healing, mental performance training, group integration, and the like.

Music or sound can be pre-recorded and delivered to one or more users 101, computer generated, or produced by one or more live musician, or any combination thereof.

In some embodiments, a live musician may also perform using bio-signal data of one or more users 101 in part or whole as a source or inspiration for musical expression.

System 100 may also be used to reconstruct scene for a user 101 at a remote location.

FIG. 19 illustrates a flow chart of a method 900 for generating a shared user experience, performed by software of FIG. 18, according to an embodiment. The steps are provided for illustrative purposes. Variations of the steps, omission or substitution of various steps, or additional steps may be considered. Blocks S910 onward are performed by processor(s) 520.

At block S910, bio-signal data is received for a plurality of users from a plurality of bio-signal sensors.

At block S920, sound data is received representing sound delivered to each of the users.

At block S930, a physiological state of each of the users is determined, based at least in part on the bio-signal data of that user.

At block S940, an environmental state associated with each of the users is determined, based at least in part on the sound data.

At block S950, using a rules engine, processor(s) 520 determine if a condition is met, based at least in part on the physiological state of each of the users and the environmental state associated with each user. Upon the condition being met, control flow proceeds to block S960. Otherwise, the condition loops.

At block S960, an associated action is executed, including generating a sensory signal.

At block S970, the sensory signal is sent to transducers to output a sensory output to each of the users based on the sensory signal.

It should be understood that one or more of the blocks may be performed in a different sequence or in an interleaved or iterative manner.

Example Mechanics

Systems and methods described herein may support a wide variety of interaction interactions to support/teach/motivate/connect a plurality of users in the context of mind+body practices or activities. The rules engine can support different interaction mechanics such as competition, collaboration, creativity expression, connection. The following are example mechanics that the rules engine can support:

Follow the leader. One or more users create a pattern that the group needs to follow. Timely and accurate replication creates feedback through output transducers. A yoga class may operate like this: teacher demonstrating, students follow along, teacher gives feedback (course correction and motivation) based on observed behaviour.

Call and response: One or more users create a pattern that the group needs to follow. Timely and accurate replication creates feedback and reward through output transducers. The call can be repeated with the response to create a bigger collective sound.

Groups vs groups: Users are divided into two or more groups. A group metric is computer for each group and is used as a basis for comparing the performance of each group in a competition, e.g. team based meditation competition.

Sprints: A challenge may be presented to peak performance in an activity for a short amount of time. For example, in a meditation sprint, users may be challenged to meditate for short sprints with relaxation and recovery moment in between. Metrics may be based on an individual, or a team for whole group based metric for motivation and/or competition. Balance sprints in yoga postures can work the same. Combined metrics such as balance, focus and heart rate may be used.

Collective goal: Group works together to achieve a goal, e.g. 5 collective minutes of calm; group synchrony for 5s. Goals can be sequenced. For e.g. yoga pentathlon: 1. Synchronized pranayama, 2. Synchronized sun salutations, 3. Collective strength posture, 4. Collective balance posture, 5. Collective meditation.

Meditation hill climb: Interval training. Meditation gets progressively harder. Challenge increases by increasing audio or visual distractions.

Mind body music jam: as described in the journey user story below, and may have more advanced features, such as: Dividing participants into instrument groups—rules engine (possible triggered by input device operated by an instructor) can queue groups to become silent, or adjust the volume of groups or individuals to orchestrate the sound (like a conductor); pulse keeper—group heartrate can set the cadence, for example, the heart rate of one person can set the rhythm—this person can change to give different people a change and to increase group cohesion.

Stop together. Each person makes a sound when they become still, as a game where people all begin in motions and the moment that one person becomes still and a sounds is made, everyone becomes still. The system can measure how quickly everyone becomes still, can see who stops last, and can vary to create different games, e.g., everyone balances, the first to loose balance, creates sound. Then everyone stops balances a soon as possible. Tally who fell first and who stopped last, or a user must maintain a rhythm or flow and the first to break creates a sound, then everyone breaks. A mode eliminates people in each round to ramp the competition to a single winner.

Swap hearts: Remote users are be paired into couples and are connected sonically using microphones and audio transducers, and experientially through their bio-signals rendered with audio and or visual feedback, Users get feedback from their own heart to get immersed in the experience, then take turns with both people getting feedback from one heart. Users can then swap hearts and or swap auditory environments. Possible additional feedbacks include: reward for synchony; highlight faster or slower heart; highlight when heart is increasing or decreasing in rate. In VR the experience of stepping in to another's body can be augmented by swapping visual fields.

Creating music to the yoga class with a user's body: Body, heart, mind, breath, makes sound allowing participants to make music through yoga. The rules engine can provide rewards for different group metrics, such as synchrony or flow. The co-created music can evolve partly based on a predetermined composition, or music composition rules.

Running groups: Create a mobile group were people are running together. Someone carries a group speaker or you listen through headphones. Music keeps a group cadence and you need to match it. If you match, you make an accompanying sound. or get a better score. Can do things like: (yoga tag, or collective goal, etc . . . ) Swap hearts. (most examples can be modified for running). Running group makes music as you run, or modifies music to reward the group as they run together, or allows for creative expression within the run.

Group Vocal tone (group OM): Accompany the Om with harmonizing sounds. Or read how your mind or heart or body is moving as you are Oming and that changes the accompaniment.

Yoga battle: Group A vs B. Group metrics that are linked to reward sound+visuals, for example, which team holds the pose the longest, the most balanced, the most flow in a sequence, the most repeatability (such as goddess pose, tree pose, supported tree pose in a group).

Meditation challenge: Challenge to find ease in the body. Stillness with subtle movement allowed from breath. Optimized to not reward for stiffness. Group finds stillness together. sounds subsides as stillness rises, or the opposite.

Hide and Seek Yoga: One person is creating sound from their movements (at the beginning they don't know either). Discover who they are through observation and copy their movements to reward through sound or other. One ending to the game is everyone following the leader. A goal can be to stay hidden and last as long as possible as the leader.

Yoga Tag: If a user is it, the user is leading. The person who is not paying attention has the lead next. Every person has a limit to how many times you can lead. Could also make it so the person who copies the best gets to lead next.

Simon Says: Conflicting demonstration and instructions. Score people or groups based on accuracy.

Meditation dodgeball: the system may throw a distraction (could be an audio or visual or other stimulus) and see who reacts the strongest. or the first, or the last, to count people out. Add cognitive challenge. disrupt (say different things, mixing up the cues). A variant allows users that are out to stay in the game by meditating to change the intensity of the next distracted that is created for the group that's still in.

Tribal Dance: Learn choreography (could be a yoga flow). Getting into a synchronized flow with other participants creates phase alignment in sound. Increasing in power with increased temporal alignment and movement similarity. There can also be an accompanying soundtrack that is reinforced by the created sounds. In a yoga practice this accompanying sound could guide and pace the practice, creating a games where using only sound, the user moves through a group practice where synchronization can be felt. In the case of a remote user, this allows the remote user to practice with a group without having to stare at a screen to follow along and feel the connection. Music can then presents a challenge, cadence, emotion. A cognitive challenge may be added by disrupting a challenge to the rhythm.

Music Recommendation

People listen to music in order to: (a) improve their performance on certain tasks (music helps us combat boredom and achieve our optimal levels of attention while driving, studying or working); (b) stimulate their intellectual curiosity (by concentrating and analysing the music we hear); and (c) manipulate or influence their own emotional states with the goal of achieving a desired mood state, e.g., happiness, excitement, and sadness. Music may also be listened to, to share a user experience. EEG can be analyzed to detect like and dislike. Some music databases may not use EEG or other bio-signal data but nevertheless have associated a mood or feeling with a particular music item, such as a song. For example, a user may be asked questions used to determine an emotional response to be associated with a piece of music. Questions may include: What was your song the last time you were really in love?; Think of a song that makes you feel sad?; Think of a song that makes you feel like dancing?; Think of a song that makes you feel inspired?; I am easy to difficult to get along with; When I feel sad: a) I listen to sad songs or b) listen to happy songs; I love karaoke or I hate karaoke; I like guitar bands or I am not a fan of guitar bands; I have a really few close friends or I have loads of friends; Body piercing can be attractive or body piercing can be a real turn-off; Life is basically simple or life is complicated; Music is all about memories for me OR I just like the music I like; My Favourite songs are sad songs OR My favourite songs are happy songs; Mess bother me OR Mess doesn't bother me; I work harder than most people I know OR I′m lazier than most people I know; I tend to worry about things OR I′m not a Worrier; I′m an Optimist OR I′m a pessimist; I hate fancy dress OR I love fancy dress; I don't get very emotional about things OR I′m a pretty emotional person; I feel uncomfortable dancing OR Love Dancing; I love meeting people and making friends OR I′m a bit shy around people I don't know; I hate it when the phone rings OR I love it when the phone rings. These questions may provide background profile of a person used to establish context for the types of songs a person likes to listen. The information can be thought of as training data that does not rely on EEG signals. The questions may determine a particular personality type of music enjoyment.

The present disclosure goes beyond merely asking questions, and associates bio-signal data (EEG brain scan data) from at least one user with audio or one or more particular pieces of music or songs that the user is listening to. This disclosure also may add EEG data of the user as additional training data to songs that have been labelled by the user as evoking a particular emotion, through the user self-reporting the emotion either through the above questions, or by tagging a song manually, and may further be used to generate a shared user experience, for example, during a meditation as described herein and with reference to system 100.

Auditory Mirror Neurons and Entrainment

There is now evidence that humans have an auditory mirror neuron system that responds both when we perform actions and when we hear the sounds of those actions being performed, and that this system facilitates empathy. Audio-visual entrainment (“AVE”) effects on the EEG are found primarily over the sensory-motor strip, frontally, and in the parietal lobe (somatosensory) regions and slightly less within the prefrontal cortex. It is within these areas where motor activation, attention, executive function, and somatosensory (body) awareness is primarily mediated. Auditory entrainment (“AE”) is the same concept as visual entrainment, with the exception that auditory signals are passed from the cochlea of the ears into the thalamus via the medial geniculate nucleus, whereas visual entrainment passes from the retina into the thalamus via the lateral geniculate nucleus. Eyes-closed AVE at 18.5 Hz has been shown to increase EEG brainwave activity by 49% at the vertex. At the vertex (with the eyes closed) AE has been shown to increase EEG brainwave activity by 21%. Successful entrainment may lead to a meditative, peaceful kind of dissociation, where the individual experiences a loss of somatic and cognitive awareness. However, it is possible for visual entrainment to trigger seizures.

Other Physiological Markers of Emotion

A variety of physiological measurements are known to have been used to detect emotional states, such as galvanic skin response (GSRe), blood volume pressure (BVP), heart rate (HR), electromyogram (EMG), skin conductivity (SC), respiration amplitude and rate (RESP), electrocardiogram (ECG), the vertical component of the electrooculogram (EOG), the tonic and phasic element of the electrodermal activity (EDA), etc.

The anterior cingulate cortex (ACC) is responsible for emotion and it may be detected by EEG. The anterior cingulate cortex (ACC) may be divided anatomically based on cognitive (dorsal), and emotional (ventral) components. The dorsal part of the ACC is connected with the prefrontal cortex and parietal cortex as well as the motor system and the frontal eye fields making it a central station for processing top-down and bottom-up stimuli and assigning appropriate control to other areas in the brain. By contrast, the ventral part of the ACC is connected with amygdala, nucleus accumbens, hypothalamus, and anterior insula, and is involved in assessing the salience of emotion and motivational information. The ACC seems to be especially involved when effort is needed to carry out a task such as in early learning and problem-solving.

There is research focused on the relation between emotional processing and frontal alpha asymmetry leading to the development of the “hemispheric valence hypothesis”. This hypothesis states that positive approach-related emotions are mainly processed in left frontal brain areas, whereas negative withdrawal-related emotions rather engage right frontal brain regions. In the EEG this is reflected by an asymmetric decrease of alpha power according to the perceived emotion, that is, a decrease of left frontal alpha power during positive emotions and a decrease of right frontal alpha power during negative emotions. There has been investigation of the trait-like frontal alpha asymmetry in the resting EEG of healthy subjects and different patient populations or the asymmetry of anterior cortical activity during stimulus induced emotional states.

Consonant and dissonant music generally may induce pleasant and unpleasant emotions in listeners, respectively. However, the impact of music on a listener is more complicated than determining dissonance and consonance. Emotionally intense music can stimulate the pleasure centres of people's brains even if the emotion is negative such as sadness or anger. Listening to emotionally intense music can relieve tension and be cathartic if a person cries for instance. Crying can relieve stress and elevate mood.

Emotionally intense music may cause dopamine to be released in the pleasure and reward centers of the brain, similar to the effects of food, sex and drugs. This makes us feel good and motivates us to repeat the behavior. The number of goose bumps observed correlated with the amount of dopamine released, even when the music was extremely sad. This suggests that the more emotions a song provokes—whether depressing or uplifting—the more we crave the song.

The choice of type of song depends on the current mood of the user, and may also be dependent on the current mood, physiological state, and environmental state of other users. Also, when we are sad some of us prefer to hear sad songs and others prefer to hear happy songs when we are sad. The most important function of music is to influence our emotional state. By keeping track of current emotional state and state after listening to music, we can gauge the degree that the music has influenced emotional state—hopefully in a positive direction, or in some embodiments, to guide a number of users to a common emotional state, and hence a shared user experience.

The present disclosure may determine the user's emotional response once, after a predetermined time has passed while playing a song, such as for example 5 seconds, or during a group experience. Optionally, the present disclosure may take multiple samples of the user's emotional response throughout playback of the song, and time-stamp any determined emotional response to correspond to time codes of the playback position of the song. One or more of the detected emotional responses of the user may then be associated with the song. Other data may also be associated with the song or used to determine the user's emotional response, such as measure of engagement (e.g. focus and entrainment with music), EEG valence, and other physiological and environmental state factors as described herein. Other sensors or other context sensors may also be used to support the emotional response determination.

One or more determinations of error-related negativity (“ERN”) may also be used to correct erroneous actions of the user.

Considerations when determining emotional response include: What is the moment to moment experience of people reaction to music?; Does person A react like person B does to the same piece on a moment by moment real time analysis?; What song do I listen to after this one?; Know—what songs do we listen to over and over. what do we skip? moment by moment allows more detailed analysis of music—vocals, bass rift, what point in the son gives us shivers; Spotify and track focus of people in music.

Issues with Categorizing Emotion

The full realm of emotion is difficult to quantify or measure in a scientifically-accurate, reproducible way. Even deciding on a language of emotion can prove difficult. This is why neuroscientists commonly use the Valence-Arousal dimensions (or VA dimensions) shown in FIG. 1 to describe emotion. In these dimensions, Valence is one axis with “approach motivation” (feelings of positivity) on one end, and “avoidance motivation” (feelings of negativity) on the other. The other opposing axis is Arousal, with high intensity of feeling on one side and low intensity on the other. Commonly-felt emotions traditionally fall within the quadrants formed by that VA matrix.

Many people wonder where emotions actually come from. Scientists are now coming to the consensus opinion that the mind and the body are more closely linked than earlier Cartesian models of cognition might have indicated. For example, the muscles associated with performing an action have been determined to move approximately seven seconds before research subjects were consciously aware of having made the decision to perform the action. In other words, by the time you recognize you are thirsty and would like to take a drink, your hand is already reaching for a glass of water. This is just one example of the complex way in which the brain and the body are linked. Not all emotion lives in the brain, but not all action lives in the body.

With EEG, recognizing the total nuance of emotion can be difficult. But it's still possible. EEG is very good at noticing changes in the brain's state. EEG measures a series of responses to stimuli that occur in the brain. EEG can recognize responses associated with these feelings: recognition; error; novelty; sleepiness; focused attention; calm.

In accordance with an aspect of the present disclosure, these detectable emotions may provide a basis for various responses described herein, however the present disclosure is not intended to be limited to these. Further emotions may also be detectable, to varying degrees of accuracy and subtlety.

One way to improve emotion detection with EEG is to add more sensors to read more data not available from the brain, or to incorporate data from other sensors on other devices that a user is also wearing. Sensors in other wearable technology devices can read things like: temperature; galvanic skin response; motion; heart-rate and pulse; muscle tension through electromyography.

These types of data can indicate involuntary physical responses from which we can deduce emotion using filtering algorithms that strain out “noise” generated from extraneous stimuli. Additional data, such as physiological state and/or environmental state, can help make a stronger case for one emotion or another. For example: an EEG might be able to sense a negative reaction to stimuli, but without contextual information from the user—either from the user's participation in an app environment, or from additional sensor data gleaned from other devices or other sensors of systems and methods described herein—it might be difficult for the system to “learn” what precipitated that negative response. Perhaps the user heard a song she didn't like on the radio, or maybe she just saw a mouse run across her kitchen floor.

User Self-Report

The accuracy of recognizing emotion can be improved when a prediction is provided to the user based on the system's analysis of their EEG. The user can reject the system's prediction and correct it with their own experience. In this way, the accuracy of the models used to predict emotion can be improved through direct user manual over-ride, using other measures of physiology related to emotion, context of the user (e.g. get information on the current activity from the user's calendar) and their behaviour (e.g. they skip over songs by artist X and they choose to listen to songs by artist Y.)

On Apps

Multiple user stories within this provisional patent refer to the use of apps by wearers of a wearable computing device of the present system. There is a specific user story related to app use below. However, it should be understood that other applications of the present disclosure are possible. All mention of “apps” may refer to applications included or provided by the system, or provided by a third-party to interface with the system.

These apps may be experienced, used, or interacted with in a variety of formats, including but not limited to: On the wearable computing device or devices; On a personal computer; On a personal mobile device, such as a phone or tablet or watch; On a website, in a browser-based application; In a vehicle equipped with the app in the dashboard or entertainment centre.

In accordance with an aspect of the present disclosure, an EEG controlled equalizer is provided that uses a control signal or test music to adjust the settings of an equalizer for a room based on the brain state of the user. The idea is to use Auditory Motor Neurons to: measure empathy in humans; use degree of empathy to drive neurofeedback among a group of humans who become empathetically synchronized to each other.

Definitions

Contextual Baseline Definition: The context of the user when using the system of the present disclosure. Context is defined by task or situation (e.g. at work or relaxing), weather, calendar appointment, time of day, location, goals of the user, who are the people with the user, external environment (e.g. room temperature, weather), and biological status of user (stressed, calm, emotional state etc.). The context is classified and the classification of the context can be used to select an algorithm pipeline to analyze and process the information received.

User Stories as Example System Architectures by Category

The following user stories may be intended to use system architecture that includes:

cloud storage of user profiles; cloud data-mining to discover new algorithm pipelines and rules for processing the EEG; and manual override of prediction by the user to help improve prediction performance. The “user stories” described herein are intended to be exemplary implementations or embodiments of aspects of the present disclosure. The present disclosure is not intended to be limited to the precise steps or features described in the user stories. In fact, aspects of the present disclosure may be intended to be implemented in a more generalized manner than that described in the user stories. For example, any reference to a specific user is not intended to be limiting.

Example Application: Music Recommendation: Recommending Sounds Based on Emotional States

An example application involves recommending sounds based on emotional states.

The Following is an Illustrative user Story for this Example Application:

-   -   Jenny wears an EEG-reading device and an audio analyzer that         contains a microphone     -   As Jenny goes about her daily tasks, the microphone continually         listens to the ambient sounds around her. (INPUT)     -   As the microphone listens to the ambient sounds, the EEG scanner         is logging Jennys emotional state (INPUT)     -   Jenny's emotional state is then logged along with the ambient         sounds into a database, either stored locally on her device or         on a remote server via the Internet on in the cloud (INPUT)     -   As Jenny listens to more and more music and visits more and more         places, an algorithm on her mobile device or on a remote server         (in the cloud or accessed via the Internet) pairs her emotional         states with the ambient sounds that are collected in her daily         travels (PROCESS)     -   Over time, a profile is built that correlates her emotional         states with the ambient sounds that accompany them (PROCESS)     -   When the profile reaches maturity a program on Jenny's mobile         device (or running on a remote server, accessed over the         Internet or in the cloud) informs her which kinds of ambient         sounds are related to which emotions (PROCESS, OUTPUT)         -   i.e. a car horn makes her anxious; a purring cat makes her             relaxed; waves crashing on the beach make her happy     -   When Jenny wants to achieve a certain emotional state, she         queries either her device or a remote server (via the Internet         or in the cloud) and the device or server will play back a sound         that's associated with her desired emotional state (happy, calm,         etc) (PROCESS, OUTPUT)     -   Jenny becomes more aware of which kinds of sounds produce which         kinds of emotional states

The Value Proposition for this user Story may Include:

-   -   There is a large market for behaviour modification tools because         people want to be able to induce different types of emotional         states (calm when anxious, etc). This process could be used in         conjunction with a meditation program or similar. Sounds are         very intimately tied to emotional states

Sensors used in this user story may include:

-   -   EEG, heart rate (stress), galvanic (stress)

Example Application: Music Recommendation: Tagging Music to Emotional States

An example application involves tagging music to emotional states.

The Following is an Illustrative user Story for this Example Application:

-   -   Brutus is shopping in a store and hears a song playing over the         PA system—Dingbat Love—while he is wearing an EEG-sensing device         and a microphone attached to his mobile device     -   While Brutus listens to Dingbat Love while shopping, his EEG         device recognizes that he likes the song very much (INPUT)     -   The microphone Brutus is wearing is continually sampling the         audio environment around him (INPUT). This audio data is either         stored locally on Brutus' device or it is stored on a remote         server via the Internet or in the cloud     -   When the EEG device detects that Brutus likes a song, this         preference is either recorded on his mobile device or on a         remote server (INPUT). Thus, a trace of Brutus' song preference         is now recorded     -   Over time, a profile of Brutus' preferred songs is built and         stored on either his mobile device or on a remote server,         accessed via the Internet or the cloud     -   When the EEG system detects that Brutus likes a song, a         music-recognition app (like Shazam) is automatically launched         (PROCESS, OUTPUT)     -   The recognition app recognizes Dingbat Love (PROCESS, OUTPUT)     -   Brutus buys the song when prompted to do so (PROCESS, OUTPUT)     -   The system suggests songs and artists similar to Dingbat Love so         Brutus can purchase additional music (PROCESS, OUTPUT)

The Value Proposition for this user Story may Include:

-   -   With the use of an EEG scanner and a microphone, music marketing         can be tailored to individual tastes (narrowcasting) rather than         to large groups (broadcasting), adding increased value

The sensors used may include: EEG, microphone.

Example Application: Music Recommendation: Tagging Sounds to a Specific Location

An example application involves tagging sounds to a specific location.

The Following is an Illustrative user Story for this Example Application:

-   -   Jaleel wears an EEG-sensing device and a microphone-enabled         listening device when he's out and about during the day     -   Jaleel listens to music while he performs his tasks     -   The microphone constantly listens to the ambient sounds of his         environment and records those sounds, either locally on his         device or on a remote server accessed via the Internet or in the         cloud (INPUT)     -   Either Jaleel's portable device or a remote server detects when         Jaleel is happy (PROCESS).     -   The device correlates the happy emotion with the sounds from his         external environment (PROCESS)     -   Jaleel's GPS sensor records the location where this correlation         takes place (INPUT)     -   Either Jaleel's portable device, or a remote server accessed via         the Internet or in the cloud, builds a profile of locations         where Jaleel hears “happy sounds” (PROCESS)     -   In a separate process, Jaleel has tagged songs on his mobile         device that he likes to listen to     -   When Jaleel reaches the GPS co-ordinates that correspond to his         “happy places”, his mobile device automatically plays the songs         he likes to hear (PROCESS, OUTPUT)

The Value Proposition for this user Story may Include:

-   -   Both listening to music and seeing specific geographical         locations (where a person had their first kiss, etc) are highly         evocative stimuli. This methodology combines those factors and         adds music to the experience, which is also highly evocative

Sensors used may include: EEG, microphone.

A map of public places that people have associated feelings about may be called Emomapping (an emotional map of a city based on sound). Customer satisfaction of sound or music: theatre manager—knowing how people feel about the quality of sound in a place. Map quietest, bird songs locations, loudest, scientists ask birders which birds they see and hear. First Kiss places share magical places in the city. Big feed of how sounds characterize a city.

Sounds are disappearing nostalgia capturing memory of sound—sounds are disappearing like rotary dial phone, leaded gas engine, old songs, old arcade games, old video game sounds. Sounds of cars like diesel—going away. Bird song. Sounds of language in a neighbourhood as demographics shift hear different languages on the sidewalk. Lose sound of bells of churches if they move. Emotional resonance to different sounds. Apply to schools, museums, think about how house sounds during different times, monitor sounds of breathing like baby monitor—apply to sound of home. The data on how people feel about sounds is the value. Value prop is we are going to make the ultimate baby monitor.

FIG. 2 shows an exemplary system view for the system of the present disclosure as optionally implemented user stories A.1, A.2, and A.3, recommending music based on emotional states.

An algorithm pipeline is chosen based on the context. In this case the context is creating a database of classified sound and its associated brain state. An algorithm pipeline ID is chosen to pre-process the EEG, extract features. The features are sent to a Brain State Classification model that outputs a brain state classification for a brief interval of time—example 1 second. The classification model could be based on prior samples of EEG generated by the user when listening to a sound. At the same time and using the same timestamps to label the EEG data, incoming audio that the user hears is classified per unit time as well using an audio analyzer that extracts features of the sound. The audio features are classified per the same unit time as the EEG brain states. These classifications are combined into the Brain state of Audio database. Examples of entries of the database are shown below. The database could be datamined for statistics or patterns. In addition, location information can be associated with the same time interval as the audio information. The Display Rules could build a colour coded map of a city or area of brain state with audio category. The Display Rules may also concatenate together shorter segments of time into an average brain state over a longer time interval. In addition, the user can do a manual override of the classification as shown by the Display Rules. The user can revise the estimate of the classification of Brain State (and Audio Classification). Based on the revised input provided the user, A Data Mining Improver can alter the methods for features extraction and the model of the Classifier. The user's input could have higher weighting when building a new model for classification.

FIG. 3 shows an exemplary entry of brain state in a database in accordance with aspects of the present disclosure.

Example Application: Music Recommendation: User Override of Music Recommendation

This example involves providing a “life soundtrack”.

The Following is an Illustrative user Story for this Example Application:

-   -   Anton frequently participates in a variety of activities that         include listening to music, like riding his bike or riding         around the city on his skateboard. He listens to music on his         mobile device. Tracks are either stored locally, on a remote         server accessed via the internet, or in the cloud     -   Anton wears an EEG-reading device while he performs these         activities and while he listens to music     -   As Anton performs his favourite activities, the EEG scanner is         constantly observing his brain states and recording the         activity, either locally on his mobile device; or in the cloud;         or on a remote server accessed via the Internet (INPUT)     -   An algorithm running on either the mobile device or on the         remote server is constantly monitoring Anton's brain state,         looking to see if he is happy or sad (PROCESS)     -   The algorithm detects that Anton is in a “happy” emotional state         while he is riding his bike (INPUT, PROCESS)     -   The algorithm then suggests a song to listen to that Anton has         previously tagged as “exciting” or “happy” (PROCESS, OUTPUT)     -   As Anton listens to music on his mobile device, he's invited to         “tag” songs with emotional states     -   This tagging can either be done manually, or automatically based         on EEG brain state data (INPUT). User can provide manual         override of the musical recommendation. This can help improve         the classification of the system and improve the accuracy of the         music recommendation engine customized to this user. This is         done through the Data Mining Improver.     -   A profile of songs correlating to emotional states will be built         up over time. This is stored in the User's Profile.     -   Based on Anton's tags, the algorithm can suggest songs to play         that correlate with other emotional states (sad, curious, etc)         (PROCESS, OUTPUT)     -   When Anton listens to music that correlates to his emotional         state, the algorithm can also suggest similar songs for purchase

The Value Proposition for this user Story may Include:

-   -   Music is very highly associated with particular emotional         states. This powerful emotional connection can be leveraged to         provide sales opportunities for new music purchases.

Sensors used may include: EEG, heart rate (stress), galvanic (stress)

FIG. 4 shows a system view of user override of music recommendation in accordance with an aspect of the present disclosure.

The Biological Signal Processing Pipeline outputs a brain state to the Music Recommendation engine of the initial brain state of the user before any audio plays. The Music Recommendation engine selects an audio track from an audio source through the audio controller. This audio track is played to the user. The user's brainwaves are continuously analysed while the audio track plays. At any point the user may decide to input a Manual Override to the system and say that this piece of music does not match my current mood which the user can input to the system. The Data Mining Improver can update the Music Recommendation rules and feature extraction and EEG classification through the User's Profile.

Manual override is an optional feature of the present disclosure.

FIG. 5 shows a table indicating user emotions before listening and after listening to a particular song.

Recommending Sound Based on user's Brain Response to Music

Johnny listens to music while wearing an EEG intelligent music system. The EEG could be embedded in the headphones, with sensors for example on the band at c3 and c4 and on the ears.

The EEG connects to a processing platform (e.g. smartphone, music player). The Processing Platform can also connect with the cloud.

Johnny's EEG and characteristics of the music are stored and analysed in the cloud.

When Johnny's brain state suggests liking of the music, for example and increase in left front activity, or an ERP, those aspects of the music can then be logged.

Those like characteristics are then compared to other music to choose music with similar characteristics which Johnny may also like, and that music is recommended to Johnny. This can also be used to compare how Johnny's brain responds to music with how other users brains respond to music, and similar reactions can trigger similar recommendations. For example “people who exhibited EEG patterns like yours while listening to X piece of music, also like Y piece of music”.

Example Application: Music Recommendation: Matching Music to Physical States

This example application involves matching music to physical states.

The Following is an Illustrative user Story for this Example Application:

-   -   Jesse listens to music extensively while performing physical         activities, like running or bike riding     -   She performs these activities while wearing an EEG-sensing and a         heart-rate-sensing device     -   A series of songs are stored on either her mobile device or on a         server accessed via the Internet, or in the cloud     -   Over time, as Jesse listens to songs on her mobile device, she         either manually tags them as something she likes to listen to         while she does exercise, or an algorithm that listens to her EEG         signal does so by correlating an EEG state corresponding to         increased physical activity to the song that she's listening to         (PROCESS, INPUT)     -   When Jesse begins to perform physical activity, this is detected         by sensors (EEG, heart rate) (INPUT, PROCESS)     -   An algorithm either stored locally on her mobile device, or         accessed via the cloud, begins playing music (PROCESS)     -   Metadata about the songs on Jesse's device contains information         about the format of the music, such as beats per minute (bpm)     -   An algorithm on Jesse's device (or accessed via the cloud)         matches the selection of music based on beats per minute to the         rate of her physical activity (as detected by heart rate         sensors, etc) (PROCESS, OUTPUT)

The Value Proposition for this user Story may Include:

-   -   Music and fitness go hand-in-glove; there are many apps that         currently exist that provide an exercise plan and allow music to         be played simultaneously. This application will allow music to         be intimately connected with the user's emotional state

Sensors used may include: EEG, heart rate.

Example Application: Music Recommendation: Detection of Songs Users Like

This example involves detection of songs users like.

The Following is an Illustrative user Story for this Example Application:

-   -   Phil is listening to a selection of new music on his mobile         device. The tracks are stored either locally on his device, on a         server accessed via the Internet, or in the cloud     -   Phil listens to his music while wearing an EEG-tracking device         which actively scans his brainwave patterns while he listens.         (INPUT)     -   As the tracks play, Phil's brainwave scanner detects his         emotional states for each song (INPUT)     -   An algorithm residing either in Phil's mobile device or located         in the cloud detects when Phil has a positive emotional response         to a song (INPUT, PROCESS)     -   The song is automatically tagged as one of Phil's favourites         (PROCESS)     -   After the song is tagged, the algorithm searches for similar         songs based on online metadata (PROCESS)     -   The algorithm suggests additional songs that Phil can purchase         based on his positive mental states

The Value Proposition for this user Story may Include:

-   -   Data about users' positive emotional states can be sent to music         producers and retailers. They can base marketing strategies         directly on listener's emotional reactions to similar music,         offering an unprecedented level of accuracy

Sensors used may include: EEG.

FIG. 6 illustrates an exemplary architecture of the present disclosure, in particular for implementing this user story. FIG. 6 shows EEG signals analyzed by the EEG Algorithm Pipeline. The pipeline extracts features from the signal. These features are used by the Brain State Classifier to output a prediction of user's brain state on a per time segment basis. An audio and or media identifier is output from for instance an digital audio player that identifies the artist and piece of music (or other media such as television show or YouTube video etc.). The pieces of media are labelled on a per time segment basis with the predicted Brain State and are stored in the Brain State Audio/Media database. Display Rules can display to the user on a per time segment basis the predicted brain states of the user to the audio or media. In addition, Display rules can process the predicted Brain States into for instance an average or a total percentage of a brain state across the entire media piece. Other Display Rules are possible. The user has the ability to manually over-ride the prediction provided through the Display Rules. This manual over-ride can be used to improve the accuracy of the Brain State classifier and or the EEG algorithm pipeline.

Example Application: Music Recommendation: Using EEG Data to Track Media Preferences

This example application involves using EEG data to track media preferences.

The Following is an Illustrative user Story for this Example Application:

-   -   Gil is an avid consumer of television programming, He wears an         EEG-reading device while he watches TV     -   As Gil watches a series of programs, his EEG monitor is         constantly monitoring his brainwave state. This information is         continually processed by either a locally-based computer or is         sent to a remote server via the internet or is sent for storage         in the cloud for later processing (INPUT, PROCESS)     -   When Gil's EEG device recognizes that he's enjoying a program, a         record of what he's watching is recorded either locally, on a         remote server, or is stored in the cloud for later retrieval         (INPUT, PROCESS)     -   The recording of the show Gil is watching can take 2 forms:     -   The system can store audio patterns from the program, which can         be stored on a local computer or on a remote server over the         Internet or stored in the cloud for later access. These audio         patterns can then be analyzed for program content manually or         algorithmically by a computer     -   Many audience-participation measurement systems rely on a         special audio frequency which encodes metadata about the program         being watched. Gil's microphone can pick up these audio         frequencies and thus immediately transmit information about         which program he's watching when his EEG scanner determines he's         enjoying the program     -   Analytics about which programs Gil enjoys watching can         automatically be sent to ratings agencies, etc     -   Analytics about which programs Gil enjoys can lead to         recommendations of new content for Gil.     -   The technology described above could also be deployed to         determine the popularity of other media, like radio and recorded         music

The Value Proposition for this user Story may Include:

-   -   Trying to determine which television and radio segments are         popular is a huge industry which consumes a lot of resources.         Plus, traditional ratings systems only measure media that people         are watching—there is currently no methodology for determining         if viewers are actually enjoying the show or if they would enjoy         similar programming. This methodology would allow for         programmers to tailor their materials specifically to what the         audience enjoys, and tailor recommendations to user's         preferences.

Sensors used may include: EEG, microphone.

Example Application: Music Recommendation: Sound Selection

This example application involves sound selection.

The Following is an Illustrative user Story for this Example Application:

-   -   Bill is listening to a selection of his music on a portable         device. The tracks are either stored locally on the device; in         the cloud; or are accessed from a remote server over the         internet. Bill is wearing an EEG-detection device while he         listens to the music     -   The system includes a pre-programmed sonic probe that is         triggered by an algorithm to be played at the conclusion of each         track     -   The “probe” consists of an odd or unusual sound (OUTPUT).     -   Bill listens as the “probe sound” plays (PROCESS, OUTPUT)     -   The EEG device records Bill's reactions as he listens to the         probe sound (INPUT, PROCESS)     -   The EEG device references Bill's brain state to determine his         reaction to the probe sound     -   If Bill records a positive response to the probe, that reaction         indicates what kind of music is going to play next (PROCESS)     -   For example, if the EEG recording detects that Bill has a         positive response to a “harsh” probe sound, then an algorithm,         either running on Bill's mobile device, or on a remote server         accessed via the internet, will instruct Bill's mobile device to         play a “harsh” piece of music next, like heavy metal or         industrial music (PROCESS, OUTPUT)     -   Similarly, if Bill's EEG detects a positive response to a         “soothing” probe sound, the algorithm will instruct Bill's         device to play a “soothing” piece of music next, like light jazz         or classical music     -   OR     -   Instead of a “probe sound”, the system instead plays a “musical         blitzkrieg”—a sequence of very short musical clips     -   As Bill is listening to the “blitzkrieg”, his EEG detector is         monitoring his brain state     -   When a positive emotional response is detected, a log is made of         which song achieves this result     -   A similar log is created when the user registers a negative         emotional response     -   In this way a profile is created indicating what type of music         creates a certain emotional response. Bill can then pick what         songs he wants to listen to based on what emotional response he         wishes to experience

The Value Proposition for this user Story may Include:

-   -   Music is very closely tied to emotional states. This algorithm         allows the user's emotional state to “pick” the next song         they're going to listen to. The system can also suggest new         songs for purchase based on the type of music the user is         enjoying, as determined by EEG measurements. Possible marketing         opportunity

Sensors used may include: EEG.

Example Application: Music Recommendation: Music Selection Appropriate to Reading

This example involves music selection appropriate to reading.

The Following is an Illustrative user Story for this Example Application:

-   -   Nancy likes to read books, magazines and other periodicals while         she listens to music     -   Nancy finds her selection of music does not always appropriately         match the “tone” of the reading material she has selected     -   Nancy now reads while listening to music whilst wearing an         EEG-detecting device     -   As she reads, her mobile device plays out a playlist of music,         which is initially random (PROCESS, OUTPUT)     -   As Nancy listens, her EEG device records her emotional response         to the song that she's currently listening to. The algorithm         that drives this process resides either locally on her mobile         device or is accessed on a remote server via the internet.         Alternatively, the brain state data is stored in the cloud for         later retrieval (INPUT)     -   If Nancy's EEG reader detects a positive emotional response to         the music she's listening to, this track is tagged in a playlist         as being suitable for reading (INPUT, PROCESS)     -   Similarly, if Nancy elicits a negative emotional response to a         particular track, the track is tagged as being unsuitable for         listening to while reading (INPUT, PROCESS)     -   As Nancy continues to read and continues to listen to music, a         playlist based on her preferences will gradually emerge. The         more she listens while reading, the more accurate the playlist         will become. (PROCESS)     -   An algorithm on Nancy's mobile device (or on a remote server)         will thus be able to predict which songs match her preference         for reading and will play those songs when she reads (PROCESS,         OUTPUT)

The Value Proposition for this user Story may Include:

-   -   Many people read while listening to music. This process makes         the two activities more compatible. There is also a marketing         opportunity to suggest purchasing new songs for reading based on         an individual's “reading profile”

Sensors used may include: EEG.

Example Application: Music Recommendation: Decreasing Stress While Driving

This example application involves decreasing stress while driving.

The Following is an Illustrative user Story for this Example Application:

-   -   Nancy likes to listen to the radio while she drives. She wears         an EEG-sending device while she drives     -   Additional sensors might include heart rate to measure stress         and galvanic skin response, also to measure stress     -   Nancy often finds music and/or talk on the radio distracts her         and makes her angry, impairing her performance behind the wheel     -   As she drives, the EEG device measures Nancy's responses to the         music she's listening to (INPUT)     -   An algorithm, running either on a local computer in Nancy's car,         or on a server which is accessed via the internet, monitors         Nancy's brain state.     -   When a negative mood is detected (INPUT) the algorithm         immediately changes the radio station or the music selection         that Nancy is listening to (PROCESS, OUTPUT)     -   Nancy's EEG monitors continually scan her brain state while she         listens to her new musical/radio selection. If a negative         emotional response is detected, the algorithm immediately         switches the selection again until a positive mental state is         achieved. (PROCESS, OUTPUT)

The Value Proposition for this user Story may Include:

-   -   The emotional state of drivers is intimately connected with         their performance behind the wheel. The Ontario Driver's         Handbook states a driver should not attempt to operate a motor         vehicle if they are in a compromised emotional position. This         tool could help drivers stay safe by ensuring the auditory         stimulation they receive is positive in nature

Sensors used may include: EEG; heart rate (stress); galvanic (stress).

Example Application: Music Recommendation: Enhanced Audio Content for Museums and Galleries

This example application involves enhanced audio content for museums and galleries.

The Following is an Illustrative user Story for this Example Application:

-   -   Martin enjoys visiting galleries and museums     -   When he visits a particular gallery, he is given an audio         headset tour guide that incorporates an EEG sensor     -   As Martin tours the museum, the audio headset tour device plays         musical selections when he stops in front of each work of art         (OUTPUT)     -   As Martin views the art, an algorithm residing either locally in         the tour device, or accessed via a remote server via the         Internet, is reading his brain state (INPUT)     -   If the algorithm detects a negative response from Martin's brain         state, the music is changed (by gently crossfading) (PROCESS,         OUTPUT)     -   This process continues until Martin's brain state gives a         positive reading (PROCESS, OUTPUT)

The Value Proposition for this user Story may Include:

-   -   Nearly every major gallery or museum around the world uses audio         tour guide devices. Adding brain state detectors would add value         and allow the museum or gallery to stand out amongst its         competitors

Sensors used may include: EEG.

Example Application: Music Recommendation: Music & Focus (L1 MR)

User stories included: 1.13.

The Following is an Illustrative user Story for this Example Application:

-   -   Morwenna wants to listen to music to help her focus on a task.         Luckily, the system of the present disclosure may help her do         this.     -   First, the system has to learn her preferences. She puts on the         devices before sitting down to work.     -   Through headphones of the system, the device plays a selection         of music while the brainwave scanner performs a scan using bone         conduction, heart-rate detection, and galvanic skin response.     -   From these sensor Outputs, the system puts together a contextual         baseline Brain State for Morwenna.     -   With the baseline in place, the system can establish differences         between Morwenna's Brain States depending on stimuli or         activity.     -   The system uses the sensors to “listen” to find which music puts         her in a relaxed Brain State.     -   Over time, the system app builds a playlist for Morwenna called         “Focus.” This is all the music that has helped Morwenna to relax         in the past.     -   Now every time she needs to focus, Morwenna can play her “Focus”         playlist.     -   Over time, Morwenna can use the system to build other playlists         bases on her emotional needs.

The Value Proposition for this user Story may Include:

-   -   This functionality allows listeners using the system device to         correlate specific music with specific emotional states.

Sensors used may include: Brainwave (bone conduction), heart-rate, galvanic skin response (stress).

Applications may include: Listening to music privately; sharing music over social networks; improving productivity.

Example Application: Music Recommendation: Matching Music to Physical States (L1-MR)

User stories included: 1.11.

The Following is an Illustrative user Story for this Example Application:

-   -   Snoop wants to go for a run—he informs his mobile device, which         is connected to the wearable computing device of the system of         the present disclosure, the wearable computing device         incorporating or connected to one or more sensors, optionally         attached to a headband, and headphones worn by Snoop.     -   Snoop activates a music-based fitness app that lives in his         wearable computing device. The app connects Snoop's music         library (either in the cloud or stored locally on his phone or         other mobile device) to his wearable computing device headband         and headphones.     -   As Snoop starts to run, his heart-rate rises.     -   An algorithm matches the beats per minute (BPM) of song to the         BPM of Snoop's heart.     -   This provides Snoop with a sense of when he is slowing down that         goes beyond the purely physical or internal. He can literally         “hear” when he slows down, like a clock that needs to be wound.         This is an easier Output for him to understand than looking down         at a heart-rate monitor, or alternative device with less legible         outputs and interfaces.     -   This more naturalistic interface allows Snoop to gain more         pleasure from running, which encourages him to run more. Now         that he runs more, he's improving his cardiovascular health.

The Value Proposition for this user Story may Include:

-   -   This functionality of the system of the present disclosure         allows the user to gain a more naturalistic understanding of         when they are slowing down during a physical activity, and         provides a more enjoyable prompt to help them speed up during         exercise. Also, it adds value to mobile music and fitness apps.

Sensors used may include: Heart-rate monitor (skin), bone conduction, temperature.

Applications may include: Physical fitness, personal training, music developers, physical therapists.

Example Application: Music Creation: Augmenting the Creation of or Listening to Music using Brainwave Activity

This example application involves augmenting the creation of or listening to music using brainwave activity.

The Following is an Illustrative user Story for this Example Application:

-   -   John is a classical music composer working on a new symphony     -   While he is working on his symphony he has an audience of         several people over to his studio to listen to his creation.         They are all wearing brainwave sensing technology     -   John plays several variations on a theme to his audience. He         wants to pick the variation that produces the most significant         emotional response.     -   John plays each variation. As the audience listens the response         from their brainwave sensing devices is recorded, aggregated and         analyzed on a server (INPUT, PROCESS). The server can either be         based locally in John's studio on in the cloud     -   The server looks for indications that the music is pleasurable         to the listeners (i.e. searches for p300) (PROCESS)     -   Alternatively, John can program the server to look for whichever         emotional response he desires (INPUT, PROCESS)     -   Alternatively John can aggregate data from other sensors (i.e.         gyroscope) to detect other physical phenomena that he wants to         elicit from the audience (like swaying to the beat or dancing in         a chair) (INPUT, PROCESS)     -   The number of desired events elicited amongst the audience is         recorded (INPUT, PROCESS)     -   John chooses the variation that elicits the greatest number of         brainwave events that he desires

The Value Proposition for this user Story may Include:

-   -   This technology allows the user to create a piece of music that         elicits particular emotional reactions

Sensors used may include: Brainwave, temperature, motion

Example Application: Music Creation: Creating a Personal Sonic “Signature” for an Individual Based on their Music Preferences

This example application involves creating a personal sonic “signature” for an individual based on their musical preferences.

The Following is an Illustrative user Story for this Example Application:

-   -   Chad listens to a selection of music on his mobile device     -   As he listens, his mobile device gathers information about which         tracks are his favourites, based on playlist frequency, etc         (INPUT)     -   Over time, the device recognizes and collates Chad's favourite         tunes (PROCESS). The tracks can either be stored locally on his         device or they can be streamed from the cloud via a service like         Rdio     -   Alternatively, a visitor to an art gallery or party can create a         sonic signature on demand     -   Jack is asked by the computer running the installation to input         3 of his favourite songs (INPUT)     -   The songs are retrieved either from a local server or via the         cloud     -   The computer/server creates a synthesis of these songs to         produce a “sonic signature” (PROCESS, OUTPUT)     -   When the algorithm has had sufficient exposure to Chad's music         to perform an in-depth analysis, the algorithm -extracts a sonic         “signature” (similar to Shazam) from each tune and then blends         them together (PROCESS)     -   This “melded” soundscape becomes Chad's personal “sonic         signature”—belonging to him and him alone (OUTPUT)     -   This “sonic signature” can be played as a ringtone or similar         (like as a chime when exchanging data with another user). The         “signature” is stored locally on Chad's device     -   If Chad is wearing Google Glass, the “signature” can be played         out as a warning tone if the AR device detects danger (i.e.         while driving, biking, etc)     -   As Chad listens to his music, a high-resolution “data map” of         his EEG responses to the track being played are recorded     -   The recording can capture other physiological states, such as         heart rate, galvanic skin response and eye tracking (INPUT)     -   This physiological information is tracked to the music on a         second-by-second basis (INPUT)     -   As the music plays, a “physiological record” of how the music is         affecting Chad can be generated (PROCESS)     -   This “data map” can be compared from person to person to see if         two people have compatible musical tastes, etc (OUTPUT)

The Value Proposition for this user Story may Include:

-   -   In a world where mobile devices and data are proliferating,         people can feel anonymous. This “sonic signature” can give a         measure of personality to the device(s) owned by the user

Sensors used may include: EEG, accelerometer (for danger sensing).

Example Application: Music Creation: Focus-Driven Musical Selection

This example application involves focus-driven musical selection.

The Following is an Illustrative user Story for this Example Application:

-   -   Linda is bored listening to her familiar mix of music on her         mobile device     -   She logs onto an internet site whilst wearing an EEG-reading         apparatus     -   She downloads a track that's been created by an artist over the         cloud. The track is either streamed from the cloud or is stored         locally on the device (INPUT)     -   As she listens to the new track, the EEG device monitors her         brainwave state (PROCESS)     -   The EEG device detects which part of the music she's paying         attention to (e.g. vocals or a specific instrument) (PROCESS)     -   An algorithm increases the “presence” of that element of the         music while “toning down” the other elements of the song         (PROCESS, OUTPUT)     -   As Linda continues to listen to the piece her focus will shift         to other components of the track. The algorithm performs the         same function again, bringing the new element of the song into         “focus” while reducing the other elements (PROCESS, OUTPUT)

The Value Proposition for this user Story may Include:

-   -   This process helps to personalize a user's listening experience         so they focus on the elements of a song they find the most         pleasurable/useful

Sensors used may include: EEG.

Example Application: Music Creation: Using Music to make a Group of People aware of the Collective Emotional State of the Group

This example application involves using music to make a group of people aware of the collective emotional state of the group.

The Following is an Illustrative user Story for this Example Application:

-   -   At a restaurant, each booth is equipped with a speaker which is         linked to a centralized database containing music.         Alternatively, the music is streamed from the cloud     -   Each diner wears an EEG-sensing headband that is in         communication with the computer/server/cloud that plays out the         music     -   As the diners eat, the EEG headband monitors their moods (INPUT,         PROCESS)     -   The moods of the diners are correlated to the type of music that         plays.     -   When the moods are pleasant, music that is harmonious and         synchronized is played (PROCESS, OUTPUT)     -   When the EEG sensor detects a foul mood, discordant music begins         to play (PROCESS, OUTPUT)     -   When the waiter/waitress passes by a particular booth, they can         immediately intuit whether the diners are happy or not.     -   If the diners are not happy, the waiter/waitress can do         something to try to rectify the situation

The Value Proposition for this user Story may Include:

-   -   Many people are too shy to say out loud if they are unhappy with         customer service. They voice their displeasure by not attending         the business again or, even worse, by recommending by word of         mouth that people they know should not attend. Using this         technology a business can nip negative attitudes in the bud as         they occur in realtime to improve service and retain customer         fidelity

Sensors used may include: EEG, heart rate (stress),. galvanic (stress).

Example Application: Music Creation: Subscription Based Music Platforms

Spotify, Rdio are examples of subscription based music platforms. Subscribers have access to the catalog of all music on the web site for a monthly fee. These companies can improve the accuracy of their recommendation engines by using standardized emotional responses to music. These web sites could get additional user information on their emotional state to help improve the classification. Classification of emotion can be highly variable and additional input from the user will help improve the accuracy of recommendations.

The Following is an Illustrative user Story for this Example Application:

-   -   Dean is going through a rough time, emotionally.     -   He wants to create a playlist that will help him “power up,” and         approach the world with less fear and anxiety.     -   Dean uses Spotify with the system of the present disclosure to         create a playlist based on his brain state.     -   He plays a series of songs on shuffle, and when he feels         energetic and confident, the system recognizes it and tags the         song with the system's emotagging system.     -   Eventually, the system helps Spotify build a playlist for Dean         that will make him feel good.     -   That playlist can follow Dean anywhere that Spotify goes.

The Value Proposition for this user Story may Include:

-   -   Value Proposition: This technology allows Dean to access an         emotionally-curated playlist of music anywhere he likes, via an         app.

Sensors used may include: EEG.

Example Application: Express Emotion Through Music: Using EEG Information, the Emotional State of the user is Displayed to Third Parties. The Emotional State of the user is Influenced by the Music they are Listening to

This example application involves using EEG information, the emotional state of the user is displayed to third parties. The emotional state of the user is influenced by the music they are listening to.

The Following is an Illustrative user Story for this Example Application:

-   -   Bono is the lead singer for a popular band.     -   His band releases a new album. Bono invites everyone to listen         to the new album during an online listening party     -   While preparing to listen to the new album, each user wears an         EEG-reading headset     -   Each user and their EEG headset are connected to the internet.         Their location is logged via IP and/or GPS (INPUT)     -   While the album plays, each user listens to it in sync and their         emotional states (happy, sad, etc) are recorded in realtime and         uploaded to the cloud (INPUT, PROCESS, OUTPUT)     -   Each user is assigned a pixel on a real-time map. The colour of         their pixel changes according to their emotional state (OUTPUT)     -   Thus each listener can access a realtime map of how they and         others are reacting to the song—it's a shifting colour, or “heat         map”. This information is shared between users via the cloud     -   After the album is over, servers in the cloud can crunch data         and provide metrics about the experience (i.e. which countries,         cities or regions thought the album was “happy” or “sad; which         geographical regions showed the greatest levels of “change” from         one emotional state to the other, etc (PROCESS)

The Value Proposition for this user Story may Include:

-   -   Music is very closely associated with emotional responses. By         collecting metrics in real time about their listener's emotional         responses, musicians can learn much more about how they are         perceived by their audience

Sensors used may include: EEG, heart rate, skin galvanic (for emotional response), GPS.

Example Application: Express Emotion Through Music: Communicating Emotional States at a Party

This example application involves communicating emotional states at a party.

The Following is an Illustrative user Story for this Example Application:

-   -   Jared is a musical performer who works in a special milieu: he         performs to a crowd directly through wireless headphones     -   These headphones are equipped with EEG-sensing technology     -   Jared performs his music; the data stream from his instruments         is transmitted, either directly or via a server located on site         or in the cloud (OUTPUT)     -   Jared's music allows people to dance together, enjoying the same         music in a synchronized fashion (OUTPUT, INPUT)     -   The headphones worn by the dancers at the event are equipped         with LEDs that are connected to the output of the EEG     -   The headphones glow according to the brainwaves generated by the         EEG. Happy emotions elicit a green glow; negative emotions a red         glow, etc (PROCESS, OUTPUT)     -   This process thus allows both the artist and the audience         members to communicate their emotional states visually to one         another

The Value Proposition for this user Story may Include:

-   -   Adding value to the nature of live performance by communicating         the emotional state of the participants as well as the         performance

Sensors used may include: microphone, EEG.

Example Application: Express Emotion Through Music: Group Musical Meditation

This example application involves group musical meditation.

The Following is an Illustrative user Story for this Example Application:

-   -   Jerome and a group of his friends gather for a collective         musical meditation session     -   Each person arrives at the group wearing an EEG sensor connected         to a headphone     -   The EEG sensors are all linked together via a central server,         existing either locally in Jerome's house or remotely, accessed         via the internet     -   The meditation session begins. Each person's EEG sensor         transmits data to the server regarding their emotional state.         (OUTPUT)     -   Emotional states are assigned a specific tone by the server         (angry=A, bored=B sharp, inquisitive=C minus, etc)     -   The server instructs each of the headphones to play the melange         of tones being generated by each of the participants (PROCESS,         OUTPUT)     -   As each member of the group listens to the sound collage, their         perception of it changes the nature of the collage, creating a         kind of feedback loop (INPUT, PROCESS, OUTPUT)     -   As the composition of the group changes (people entering,         leaving the group) or as the ambient sounds in the environment         change, therefore changing the feedback pattern of the group and         changing the sound collage     -   External sounds picked up by the microphone are capable of         changing the mood dynamic of the group as well, enhancing the         collage of sounds that's being produced (INPUT, PROCESS, OUTPUT)

The Value Proposition for this user Story may Include:

-   -   This process will allow us to create a new dimension in         mindfulness meditation, enabling users to create their own         “mindscapes” and to foster mutual cooperation

Sensors used may include: EEG, microphone.

Example Application: Express Emotion Through Music: Visualizing an Emotional Connection to an Instrument

This example application involves visualizing an emotional connection to an instrument.

The Following is an Illustrative user Story for this Example Application:

-   -   Spike is a famous violinist who plays a modified instrument         equipped with LEDs and a wireless connection to an EEG-sensing         apparatus, which Spike wears     -   Both the EEG sensor and Spike's violin are wirelessly linked to         a server in the concert hall     -   As Spike plays during the concert, his emotional state is read         by the EEG-sensing apparatus (INPUT)     -   His emotional state at each moment is read by the apparatus and         is transmitted to the server (OUTPUT)     -   The server continually analyzes the output of the EEG device         (PROCESS)     -   The server instructs the LEDs in the violin to change colour in         response to Spike's changing emotional state (OUTPUT)     -   Spike can let these changes flow naturally or he can         deliberately cultivate emotional states to change the colour of         his violin (OUTPUT)

The Value Proposition for this user Story may Include:

-   -   Enhancing the value of musical performances by having the artist         share his brain state information with the audience

Sensors used may include: EEG.

Example Application: Express Emotion Through Music: Converting Emotions into Music

This example application involves converting emotions into music.

The Following is an Illustrative user Story for this Example Application:

-   -   Jim and Sally are in a relationship. They own a system that         consists of two EEG-sensing devices they wear when they are         apart.     -   The EEG devices constantly measure both of their brainwave         activity. The data is collected and mutually shared, either         directly over the internet via their mobile devices or via a         mobile server in the cloud (INPUT)     -   As Jim and Sally go about their activities during the day, both         of them generate a continual stream of data from their         EEG-sensing devices, which correlate to their emotional states         and their levels of focus     -   These emotional states are compiled and analyzed by an algorithm         residing either on their mobile devices or on a remote server         accessed via the Internet (PROCESS)     -   As this detection process reaches maturity, both Sally and Jim         will have worn their EEG-reading devices while listening to a         large number of music tracks     -   Each track in their library will be tagged with an emotional         state (i.e. “happy”, “sad”, etc) based on the EEG responses         generated during listening (PROCESS, OUTPUT)     -   The more they listen, the larger and more accurate the database         of tagged music becomes     -   Jim and Sally's emotional states are correlated to the tags on         the music selections in the song libraries on their mobile         devices. (PROCESS)     -   A continual stream of “mood-focused” music then plays based on         Jim and Sally's constantly-fluctuating emotional states (OUTPUT)     -   Even though they may be separated by large geographical         distances, either Jim or Sally can check their partner's “mood         music stream” to see how the other is feeling at any given         moment (OUTPUT)

The Value Proposition for this user Story may Include:

-   -   This technology will allow couples (or friends) to experience         new levels of intimacy and will allow for improved harmony in         their relationships

Sensors used may include: EEG.

Example Application: Behaviour Modification: Using Biofeedback to Improve Driving Performance

This example application involves using biofeedback to improve driving performance.

The Following is an Illustrative user Story for this Example Application:

-   -   Caleb drives many kilometres each day as part of his job (sales)     -   Caleb's insurance providers will offer him discounts if he stays         alert and attentive while on the road     -   Caleb likes to listen to music. He also wears brain-sensing         equipment while he's on the road (a wearable computing device of         the present disclosure or similar)     -   Caleb's sensors monitor his brain state while driving (INPUT)     -   If the detector determines that Caleb is beginning to lose         focus/become drowsy, the brainwave detector will instruct         Caleb's car to change the music he's listening to (PROCESS,         OUTPUT). Information about which music can be accessed can be         stored either on a local device in the car or in the cloud     -   Brainwave detector monitors Caleb to ensure the new music has         changed his focus/attention levels (PROCESS)     -   If Caleb's focus starts to drift again, brainwave sensors will         detect this and change the music selection again until a stable         attention pattern is achieved (PROCESS, OUTPUT)

The Value Proposition for this user Story may Include:

-   -   There are many industries (manufacturing, transportation, etc)         in which a consistent level of focus is desired, especially when         workers listen to music. This process will provide a consistent         methodology for maintaining focus levels

Sensors used may include: EEG, gyroscope (to detect head position if getting drowsy).

Another case could be to improve focus through the selection of background music. The user is given feedback on their state of focus and concentration. The user can insulate themselves from the external environment as well become aware of emotional issues that arise within themselves. The background music can be changed to help improve these factors.

GOAL: FOCUS—how much concentration and distraction—measure of how well we are doing. System tries different variations of background music. User can emphasize what they like (turn up volume)—this provides information from the user as to their preferences. User preferences can also be learned when they skip over a song therefore the system learns which songs are not suitable for focus and concentration. Also need to learn if the user turned down the volume because something else happening in their environment. Example: user is sitting at computer and being tracked with front facing camera whether user is focussing on-screen- or on the p hone. A thinking profile can also be chosen to optimize performance. The profile can be think for 5 minutes, rest for 3, think for 10 minutes etc. The background music is synchronized to the profile.

Example Application: Behaviour Modification: Biofeedback for Mindful Speech

This example application involves biofeedback for mindful speech.

The Following is an Illustrative user Story for this Example Application:

-   -   Johnny wears an EEG-sensing device equipped with a microphone     -   The device and microphone both communicate with a computer,         either in Johnny's mobile device or located on a remote server         that's accessed via the Internet     -   An app stored in Johnny's mobile device records Johnny's voice         and his emotional state (as captured by the EEG device)     -   The app creates correlations between his brain state and his         vocal tone     -   As Johnny uses the app more and more it becomes able to predict         his vocal tone based on his brain state     -   The app will alert Johnny, either through audio feedback or         through a vibrating mechanism, when his current speech may not         be received as he intends it to be     -   Based on this new level of awareness, Johnny can choose to alter         his speech patterns or, alternatively, he can choose not to         speak at all—depending on his current context     -   Johnny's EEG sensor can be connected directly, either via the         internet or via the cloud—to another individual's EEG sensor,         which is also connected to an app running on that person's         mobile device, or which is resident on a server accessed via the         internet     -   This connection facilitates Johnny's emotional state directly to         the other user so the other user can best judge Johnny's         emotional state with as little confusion involved in the process         as possible

The Value Proposition for this user Story may Include:

-   -   Creates stability for individuals with poor affect judgment         (Asperger's spectrum, etc or autism) that will enable them to be         better judges of other people's emotions. Can serve as a         valuable therapeutic tool

Sensors used may include: EEG, microphone.

As another example, Johnny and Brenda tend to have heated arguments at work, so they hire a conflict resolution expert who uses EEG-sensing devices equipped with a microphones

Johnny and Brenda wear the devices which both communicate with a computer setup where both people can see a display of each other's state.

After calibrating, Johnny and Brenda are asked to discuss a sensitive topic, and the devices record their voice and emotional states (as captured by the EEG device).

The application creates correlations between brain state and his vocal tone and displays their emotional states during the discussion to the other speaker.

The application alerts Johnny and Brenda in real-time, either through audio or visual feedback, when their current speech may not be received as they intend it to be.

Based on this exercise, Johnny and Brenda learn to alter their speech patterns and when not to speak at all—depending on each other's emotional state.

This exercise facilitates a better work relationship between Johnny and Brenda by training them to recognize each other's emotional states emotional and learn to be more mindful of their actions and speech when working with each other. judge

The value proposition for this system architecture may include many other use

cases: individuals with poor affect judgment (Asperger's, autism) that may enable them to be better judges of other people's emotions. This may serve as a valuable therapeutic tool. Further example may be preparing for a presentation or speech with a direct read on audience emotional states and using that to improve the presentation/speech.

Sensors used may include: EEG, microphone.

Example Application: Using music to change your Mood: Using Music to Change your Mood (MR)

This example application involves using music to change your mood.

The Following is an Illustrative user Story for this Example Application:

-   -   Helo wants to improve his mental state, to become “cheerier”     -   Helo instructs his mobile device that he wants to change his         mood to something more positive     -   The mobile device responds, initially, by playing a random         selection of tracks. The music data is either stored locally on         Helo's device or is streamed via the Internet     -   An EEG-measuring device (e.g. the wearable computing device of         the present disclosure) keeps track of Helo's emotional state         while listening to the music selection     -   A shift towards a positive emotional state is tracked by the EEG         device. Helo's mobile device records the song as being         “emotionally positive”     -   Over time, the EEG device builds a profile of “positive songs”         based on EEG measurements. This “mood data” will be stored         locally on Helo's device but for analysis it can be sent to the         cloud     -   Additional indicators of positive mood may be motion-based         (dancing) or increased heart rate (excitement)     -   As Helo plays more songs the algorithm that determines whether a         song is “positive” becomes more and more accurate. Similar to         Google Translate, the analytics can be performed on the cloud     -   After a while the mobile device will automatically play         “positive songs”     -   The EEG reader (and/or other sensors) will continually check the         validity of the algorithm's predictions in case Helo's response         to the songs change

The Value Proposition for this user Story may Include:

-   -   Improvement of emotional states is a huge market (brain training         apps like Lumosity, etc). EEG-based monitoring of how music         affects mood will be a valuable tool to exploit this market         since music and emotional states are closely correlated

Sensors used may include: EEG, heart rate, stress (galvanic), motion (gyroscopic).

Example Application: Therapeutics: Using Music and Brain Scan Technology to Aid in Injury Recovery

This example application involves using music and brain scan technology to aid in injury recovery.

The Following is an Illustrative user Story for this Example Application:

-   -   Phil has sustained severe injuries to his right leg in a car         accident     -   To regain the use of his leg, Phil undergoes a few hours of         intensive physiotherapy each day while wearing a brainwave         scanning device     -   Phil likes to listen to different types of music while he works         on his recovery, including classical, jazz and rock     -   Each of these different types of music affects Phil's mental         state in a different way. The physiotherapy staff monitors these         changes because they want to optimize Phil's recovery     -   Throughout Phil's treatment, a profile of how he responds to         each type of music is recorded by a computer     -   For example: jazz relaxes him; rock energizes him; classical         improves his level of focus     -   As Phil is exercising, his brain state is monitored by the         clinic's staff     -   The clinic staff change the type of music to match the desired         treatment outcome     -   i.e.: if they want Phil to be relaxed, jazz is played; if they         want to improve focus, classical music is played     -   If Phil's responses to the music change over time, these changes         are logged by the computer and treatment options are varied         accordingly

The Value Proposition for this user Story may Include:

-   -   Therapeutic treatment options for a variety of conditions are as         varied as the people who are seeking therapy. Music is a         well-documented therapeutic tool. By tracking specifically how         different people respond to different types of music a         highly-personalized, individual treatment profile can be         generated that changes according to the therapeutic outcome.

Sensors used may include: Brainwave, stress (galvanic response), body temperature, movement (gyroscope).

Example Application: Therapeutics: Musical Psychotherapists

Use music for emotional healing. They can improve their practice by obtaining more objective emotional data in terms of before, during and after playing of music. The therapist can more quickly determine which music selections are having the biggest impact on their patients.

Music therapy typically comprises of an a therapist and a patient or group of patients. Patient plays music on instruments and non-instruments alike to gain emotional contact with his internal experience. The therapist is there to guide him, hold space, or talk about the insights and experiences that arise for the patient. In EEG enabled Musical therapy, the music can be made directly from a patients EEG activity. For example, brain input can go into a midi or other musical controller such that brain activity maps to sound creation. For example low frequency brainwaves can be mapped with low frequency sounds, and high frequency brainwaves can be interpreted to produce high frequency sounds. Or brain activity can control an aspect of the sound, like pitch or volume. Lighting can also be mapped to brain state and co-vary.

A group of patients can each play music created with their own brain state, and play in concert with one another. For example, when the players synchronise brainwaves, new effects in the music can be created.

The therapist can play alongside the patient as well. For example with reward when the patient and therapist are in synch with their brain activity, for example they could be in phase, cohere, or same frequency.

Rapport between patient and therapist has been known to be highly beneficial to the therapy. Also, an EEG system could detect the mood of a patient, via their brain activity. Sound could be mapped to mood, so the patient can “hear” their changing moods, or work with sound to alter mood. For a simple example a sad mood could be mapped to low sounds, and a happy mood to high sounds, and patient would practice changing mood by changing sound.

Example Application: Therapeutics: Detection of Brain Events using EEG Technology

This example application involves detection of brain events using EEG technology.

The Following is an Illustrative user Story for this Example Application:

-   -   Jenny works as a neuroscientist in a hospital. She uses         EEG-sensing technology to obtain new therapeutic outcomes for         her patients     -   Jenny applies EEG sensors to her patients to obtain neurological         indicators of their brain health     -   These recordings take the form of a “brain album”—they consist         of EEG recordings of various neurological conditions, such as         stroke, seizure or migraine (INPUT, PROCESS)     -   Jenny or her institution develop an app that can be downloaded         by patients with various brain disorders     -   This app can be run continuously by patients. By using this         program, patients can “hear” when they are likely to experience         a traumatic brain event, such as a seizure or a stroke (OUTPUT)     -   This information is transmitted to them via a unique auditory         signature through a pair of headphones. Each “brain event”, such         as a stroke or seizure, is correlated to a specific auditory         signal (PROCESS, OUTPUT)     -   When negative brain events are not occurring, Jenny's patients         can listen to music that they enjoy. But the EEG-based “early         warning system” is continually monitoring their brain activity         to sense the presence of oncoming negative brain events.         (PROCESS)

The Value Proposition for this user Story may Include:

-   -   Neurological disorders are common and place a large economic and         social burden on society. In addition, detecting brain trauma         early is a key method to reduce the destructive impact of the         events. By deploying EEG technology, early detection of negative         brain events can be improved

Sensors used may include: EEG.

Example Application: Therapeutics: Detection and Enhancement of ASMR (Autonomous Sensory Meridian Response)

This example application involves detection and enhancement of ASMR (autonomous sensory meridian response).

The Following is an Illustrative user Story for this Example Application:

-   -   Laura is an individual who experiences ASMR. This means she         experiences a “tingling” sensation along the spinal column when         she hears particular sounds     -   Examples of sounds that trigger this sensation include         whispering, clicking, crinkling of soft materials, lip-smacking         or breathing     -   Laura downloads an app onto her mobile device that is connected         wirelessly to an EEG-sensing device that she wears. Laura also         wears a microphone     -   The app is programmed with information that conveys that Laura         enjoys the ASMR experience—she finds ASMR trigger sounds calming         and soothing (PROCESS)     -   When the app is running, it listens to Laura's ambient         environment continuously (INPUT, PROCESS)     -   In the process of listening, the app assists Laura in finding         people in her environment who will stimulate the ASMR experience     -   This stimulation is based on patterns of speech; accents; volume         levels of voices (INPUT, PROCESS)     -   When the app detects a sound pattern that is likely to trigger         an ASMR response in Laura's general environment, the app         responds by “turning up the volume” on that particular         individual (INPUT, PROCESS, OUTPUT)     -   The result is that even though the ASMR-triggering sounds may be         quiet in the general environment, they sound loud and clear to         Laura

The Value Proposition for this user Story may Include:

-   -   ASMR is a therapeutic process that makes people relaxed and         provides comfort. An app that specifically selects sounds that         are likely to induce this response has therapeutic value

Sensors used may include: EEG; microphone; galvanic skin response (to detect “tingling” when ASMR response is achieved.

Example Application: Therapeutics: Depression Amelioration through Positive Brainwave Reinforcement

This example application involves depression amelioration through positive brainwave reinforcement.

The Following is an Illustrative user Story for this Example Application:

-   -   Joseph suffers from depression     -   He wears an EEG-measuring device as well as a headphone     -   He has an app loaded onto his mobile device that detects his         brainwave activity while he listens to music (INPUT, PROCESS)     -   This app is designed to sense alpha waves—a signifier of         positive mental health (PROCESS)     -   As Joseph listens to music on his mobile device, the app         interfaces with his EEG-sensing device, scanning his brainwaves         for the presence of alpha waves (INPUT, PROCESS)     -   Over time, as Joseph listens to more and more music, the app on         his mobile device will be able to detect which songs are capable         of inducing alpha waves (PROCESS)     -   The more music Joseph listens to on his mobile device, the more         accurate the profile of his alpha-wave-inducing playlist becomes         (INPUT, PROCESS)     -   Joseph listens to songs tagged with alpha-wave inducing         properties to boost his mood when he's feeling low (OUTPUT)     -   Joseph's headphones are equipped with bright LEDs in close         proximity to the ears     -   The lights glow when Joseph is listening to his alpha-boosting         playlist (OUTPUT)     -   Research suggests that shining light into the ears can increase         serotonin levels (associated with depression amelioration) after         8 minutes of exposure

The Value Proposition for this user Story may Include:

-   -   Listening to music is a proven method to help people tackle         depression. The methodology described above can maximize the         benefits of music-based therapy by using specific songs that are         shown to have a therapeutic benefit

Sensors used may include: EEG, microphone.

Example Application: Therapeutics: Assistance for the Blind using EEG Technology

This example application involves assistance for the blind using EEG technology.

The Following is an Illustrative user Story for this Example Application:

-   -   Ray is blind     -   He is afraid that he will miss key emotional cues from other         people because of his lack of sight     -   He is also concerned that people he speaks to (doctors, nurses,         service staff) don't listen to him as well as they should     -   As part of his care regimen, people who frequently associate         with Ray are encouraged to wear EEG-sensing devices     -   Ray wears an EEG-sensing device too, as well as a headphone     -   The outputs from the EEG-sensing devices from the people who are         around Ray are transmitted into Ray's mobile device, either         directly, or through a remote server accessed via the Internet         (INPUT)     -   Using an app installed on his mobile device, Ray is able to hear         changes in the brain state of the people who surround him         (PROCESS, OUTPUT)     -   The changes in brain state are interpreted by the app running on         his mobile device as a series of tones (OUTPUT)     -   When the people who surround Ray have their brain state change         when Ray is talking to them, a tone will play in Ray's headset         allowing him to detect when the change occurs (PROCESS, OUTPUT)     -   In addition to EEG-sensing devices, the people who surround Ray         can also be compelled to wear additional sensors that track         changes in mood (such as heart rate and eye track). When these         sensors pick up changes that vary from baseline, Ray will hear a         musical tone in his headphones

The Value Proposition for this user Story may Include:

-   -   The implementation of this technology will allow blind and         visually-impaired people to participate more in the cultural         milieu that surrounds them. They will be able to feel more         empowered, less ignored and can have more meaningful         conversations with their caregivers. The same technology could         be used to improve the lives of people suffering from         prosopagnosia (face blindness) or autism spectrum disorders

Sensors used may include: EEG; heart rate; eye track; galvanic skin response.

Example Application: Therapeutics: Support of Brain States Conducive to Sleep

This example application involves support of brain states conducive to sleep.

The Following is an Illustrative user Story for this Example Application:

-   -   Billy puts an EEG-sensing device before going to sleep at night     -   Billy's mobile device plays music (OUTPUT)     -   The EEG-sensing device tracks Billy's brainwave activity while         he listens to music in bed (INPUT, PROCESS)     -   An app installed on Billy's mobile device tracks Billy's sleep         patterns as he listens to the music tracks on his mobile device         (PROCESS)     -   Songs that induce restful sleep are tagged on his device as         being useful for sleep (PROCESS)     -   The more Billy listens to music on his device, the more         accurately the app can predict which songs are likely to induce         sleep, based on EEG-derived information (PROCESS)     -   A feedback mechanism is enabled which allows the system to         evolve to match Billy's preferences as they change as he listens         to more and more music (PROCESS)     -   As the music plays while Billy is in bed, the EEG device is         monitoring his brain state and delivering that information to         the app on his mobile device (INPUT, PROCESS)     -   When the EEG device detects that Billy is approaching a sleep         state, the app automatically fades the music volume down         (PROCESS, OUTPUT)     -   When the EEG device detects that Billy has fallen asleep, the         app on his device fades out the music entirely (PROCESS, OUTPUT)     -   If Billy wakes up at any point during the night, the EEG device         will detect this change and music that supports sleep will begin         to play again (PROCESS, OUTPUT)     -   This process will repeat in a loop as many times as is necessary         for Billy to sleep peacefully throughout the night (PROCESS,         OUTPUT)

The Value Proposition for this user Story may Include:

-   -   Insomnia and sleep disorders are a significant health problem         for a significant percentage of the population. A drug-free way         to remedy the problem has significant market potential

Sensors used may include: EEG.

As another example, a user may be playing music during the day time to train sleep at night. As an illustration, one plays Music containing 13-15 hz sounds. 13-15 hz is the frequency of sleep spindles, they arise during stage 2 sleep and indicate the onset of sleep. Research has shown that training sleep spindles from areas including the sensory motor cortex during the day leads to improved sleep latency, and also improved declarative memory the next day. The user can listen to Music that contains 13-15 hz binaural beats to entrain the brain. The user can listen to music and be wearing an EEG with sensors at, for example c3 and c4 (in 10-20 system), and when the user produces a 13-15 hz frequency the music will adjust as a reward for the listener, thereby entraining 13-15 hz spindles.

Example Application: Assessment of Human Emotional Response to Music: Emotional Appraisal Using Stimuli

The Following is an Illustrative user Story for this Example Application:

-   -   Sigmund is a member of the Canadian Armed Forces. He applies for         a promotion, and is introduced to Dr. Jones.     -   Many people who have PTSD will have triggers such as certain         sights, sounds, or smells often around the anniversary of the         trauma—See more at:         http://www.bandbacktogether.com/ptsd-resources/#sthash.tekWKE6E.dpuf     -   Dr. Jones wants to do a psychological assessment of Sigmund as         part of the interview process for the promotion, which will         require a cool head in high-stress situations, and split-second         decision-making. Dr. Jones is concerned that Sigmund's         experience in Afghanistan has led to PTSD. If it has, and if         Sigmund earns the promotion, Dr. Jones will need to help Sigmund         manage his condition so that he can be an effective leader and         communicator.     -   Dr. Jones fits Sigmund with a brain-sensing wearable computing         device of the present disclosure or suite of devices.     -   The assessment consists of listening to certain sounds in the         headphones of the wearable computing device while brainwaves are         read by the device(s). Dr. Jones needs to understand Sigmund's         response to that audio stimulation, as a means of understanding         how he will react in specific situations. The stimuli involves         loud noises, loud music, voices talking over each other, raised         voices, and other common audio triggers to patients with PTSD.     -   As Sigmund experiences the stimuli, Dr. Jones watches how         Sigmund's brain responds via the wearable computing device. Dr.         Jones takes note of eye-tracking (EOG) and Steady-State         responses to stimuli to work up a diagnosis.     -   After the assessment, Jones is provided with an assessment         report detailing Sigmund's emotional responses to the various         stimuli and what that means about his emotional state and         intelligence.

The Value Proposition for this user Story may Include:

-   -   The system of the present disclosure offers users access to         real-time data on emotional and psychological responses to audio         stimuli.

Sensors used may include: emotional responses detected in real-time while user engages with stimuli.

Applications may include: Therapy, on-the-job assessment, testing, academic/pedagogical, mental health arena, testing for specific.

Example Application: Machine Maintenance: Funny car sounds alerts mechanic

The Following is an Illustrative user Story for this Example Application:

-   -   Wendy is wearing a wearable computing device of the present         disclosure. She is driving down the highway when all of a sudden         her car starts making a funny sound. Brainwave sensors pick up         the salient P300. This informs the car's onboard computer to log         the associated engine measures with the perceived sound. The         sound may be intermittent.     -   Wendy's mechanic can be alerted and auto manufacturers can use         the knowledge to improve diagnostics and automobile performance.

The Value Proposition for this user Story may Include:

-   -   improved ability to diagnose intermittent and or persistent         sounds. Early warning of changes in sound characteristics that         can prevent expensive auto repairs.

Sensors used may include: auto sensors; EEG sensors.

Example Application: Filtering of Content: Ad Blocker

The Following is an Illustrative user Story for this Example Application:

-   -   Fred is listening to the radio via his wearable computing         device(s) of the present disclosure when his reverie is         disturbed by an annoying ad. Fred is interested in some ads, but         there are others that he prefers to ignore.     -   The system of the present disclosure has a context-dependent         Brain State profile of how Fred's brain reacts to regular         stimuli as a baseline. When Fred doesn't like something, he         registers a negative reaction via the system.     -   The system pattern-matches Fred's response against the specific         ads that Fred was hearing at the time.     -   The system of the present disclosure filters out the ads that         Fred prefers not to hear.

The Value Proposition for this user Story may Include:

-   -   This technology allows users to dictate which ads they hear, and         provides advertises with rich data on what ads they like, as         well as how their brains respond to each ad. Eventually, this         leads to better, more enjoyable advertising that people actually         enjoy listening to.

Sensors used may include: microphones, EEG.

Applications may include: radio, television, and web.

Example Application: Filtering of Content: Selective Eavesdropping

The Following is an Illustrative user Story for this Example Application:

-   -   Winona is in a cafe wearing a wearable computing device of the         present disclosure or suite of devices. It can sense brainwaves         and can focus microphones to tune in a sound in frequency and         direction.     -   She hears a speaker (or a voice, or a sound) over the din that         her brainwave sensors detect as being pleasant. Her headphones         begin tuning in the listener and Winona P300 response is used to         bring the target of her attention into focus so she can listen         in selectively to a conversation.     -   Depending on Wnona's responses and her contextual baseline Brain         State, Winona can choose to filter out the sounds that she         doesn't like. Or she can make sure that she doesn't hear         conversations that she has no interest in.

The value proposition for this user story may include: People in public spaces don't always remember that their conversations can be overheard. This would give their neighbours the opportunity to screen out their conversations, or “listen closer” if the conversation is really good.

Sensors used may include: EEG, microphones, baffles that guide and channel sound.

Applications may include: Gossip blogging, social networking, interviews, journalism.

Example Application: Filtering of Content: Selective Hearing (L3)

The Following is an Illustrative user Story for this Example Application:

-   -   Dick literally never listens to Jane.     -   Dick programs the system of the present disclosure to listen for         when Jane is angry, based on her blood pressure, temperature,         and other available inputs, available from Jane's freely-shared         system data.     -   Dick programs the system to play white noise over Jane when         she's angry. Dick doesn't hear her anger.     -   Their relationship finally snaps when Jane realizes he's been         using the system to literally tune her out.

The Value Proposition for this user Story may Include:

-   -   The ability to tune out the things or people who cause us         distress is a double-edged sword. On the one hand, we can avoid         hearing things like hate speech, or curse words, or bullying         comments that undermine our sense of self. On the other hand, we         could literally tune out the people around us. Depending on what         the user wants, this can go a variety of ways.

Sensors used may include: Skin temperature, blood pressure, frontal lobe sensors.

Applications may include: Mediation, negotiation, avoiding “triggering” conversations, screening out angry people in a customer service or public service context (angry people in line at the DMV, etc.).

Example Application: Filtering of Content: Whispering Gallery

The Following is an Illustrative user Story for this Example Application:

-   -   Alex is slowly losing her hearing. She has trouble discerning         one person's voice from another's in a crowd. Unfortunately, she         works as a journalist at City Hall and often needs to pay close         attention during press conferences and scrums with the Mayor,         when things can get noisy.     -   Alex purchases the wearable computing device of the present         disclosure to help her filter out the sounds she doesn't want to         hear.     -   Alex spends time listening to specific sounds, and the system of         the present disclosure reads her reactions to understand the         things she doesn't want to hear (like the sound of a camera         shutter, for example, or the sound of someone else's text         messages arriving).     -   Alex also practises with the system to understand whose voices         she should be hearing during a scrum. The system uses         microphones to record entire press conferences, but screens out         other reporters' voices so that Alex can concentrate. This way,         she can re-play the entire conference if she wants to, but she         doesn't feel the same stress and frustration that she used to         during a press conference.

The Value Proposition for this user Story may Include:

-   -   This technology would enable the hearing-impaired to listen only         to what they needed to hear, while also recording an entire         conversation that they might otherwise have missed. Similar to         real “whispering galleries”:         http://en.wikipedia.org/wiki/Whispering_gallery

Sensors used may include: Microphones, gyroscopes, motion detection.

Applications may Include:

-   -   Noisy environments, press galleries, crowded environments.

Example Application: Filtering of Content: Personal Channels/Brainwave as Tuner/The Sound of Long Distance Relationships (L3)

The Following is an Illustrative user Story for this Example Application:

-   -   Audrey is in a long-distance relationship with Humphrey. She         lives on the West Coast, and he lives on the East Coast.     -   One of the ways the two of them stay connected is for each of         them to “tune in” to the sounds that each of them enjoys hearing         in their local environments. For example, Audrey likes to hear         the sound of the native birds in Humphrey's neighbourhood when         he wakes up, even though he wakes up three hours ahead of her.     -   The system of the present disclosure logs when Humphrey wakes up         by analyzing and interpreting his brainwaves, and begins         recording the sounds that Humphrey is hearing.     -   The system logs the sounds that Humphrey appears to enjoy. It         takes note of his brainwave responses and also measures the         movements of muscles in his face and where his eyes go, via         eye-tracking.     -   When Humphrey listens closely to a sound, the device “learns”         his response and begins recording the sound.     -   Each day, the sounds are sent to Audrey's wearable computing         device of the present disclosure. Audrey sets a default in the         headphones to make sure that these sounds are her wake-up alarm.         Each morning, Audrey hears the sounds that Humphrey heard upon         waking, when she wakes up.

The Value Proposition for this user Story may Include:

-   -   Finding out what sounds people focus on during the day is         important to understanding emotion, productivity, perception,         and intelligence. With information like this, people can grow         closer and understand each other better using raw data, rather         than interpretation.

Sensors used may include: Microphones, eye-tracking, sensory strip, galvanic muscle response.

Applications may include: Long distance relationships, surveillance, eavesdropping, journalism.

Example Application: Social Networking Through Musical Preferences: Sharing the Same Tastes in Music

Finding out the music that someone else likes seems to give you a lot of information about them quickly. For example, college students getting to know each other over the internet may be more likely to ask about music preferences than about all other categories of conversation topics combined. Further, knowing someone's music preferences may allow students to do a reasonable job of predicting some of the new person's personality characteristics and values. Unsurprisingly, people expressed that they liked a new person better when finding that they shared the same musical taste than when they did not.

This example application involves sharing the same tastes in music.

The Following is an Illustrative user Story for this Example Application:

-   -   Finding a partner online is a difficult task because of how easy         it is for one to create a persona that is not truly         representative of their personality. What is necessary are more         honest online personae, with real information on a person's         tastes.     -   Darla goes on-line to a dating web site. She is worried about         “being herself,” but she's also ready to get serious with her         online dating profile.     -   While wearing a wearable computing device of the present         disclosure, she listens to music that is selected by the dating         web site to be representative of certain genres. She listens to         tunes she has never heard before, as well as old favourites. She         listens to multiple genres, some of which she never really         listens to and others which she hasn't listened to in quite some         time.     -   The system of the present disclosure records all her Brain State         responses to the music, brainwaves detected from the auditory         cortex, pre-frontal cortex and other parts of the brain involved         in processing and creating emotional responses to music. In         addition motion detection, and eye-tracking are analyzed to         determine if the user is dancing or moving to the music adding         additional evidence that she is having an emotional response to         the music.     -   The system records and stores her preferences of music based on         frequency of song choice, dancing, and Brain State.     -   This generates an Output that is much like a Profile. The         Profile has all her music preferences (even the ones she's not         consciously aware of, like songs she enjoys but hasn't purchased         yet).     -   Her Profile is used to match to the preferences of other daters         on the web site.

The Value Proposition for this user Story may Include:

-   -   The system can capture people's authentic reaction to music.         Taste in music is often an important predictor and selection         criterion for people looking for a partner.

Sensors used may include: brainwave sensing headband.

Applications may include: Online dating, music promotion, music label/A&R promotion.

Example Application: Social Networking Through Musical Preferences: Musical Genre Definition Through Emotion

This example application involves genres of emotion

The Following is an Illustrative user Story for this Example Application:

-   -   Blingo is a music executive who is about to release a new         artist.     -   Blingo doesn't know exactly how to determine the genre of this         new experimental music; he just knows that he wants to promote         it and help it find the right audience who will appreciate it.     -   Lucky for Blingo, traditional genres are a thing of the past.         Thanks to the end of radio, musical artists are less about genre         and more about branding. He just needs to develop this artist as         a brand and hope that brand finds an audience.     -   Blingo submits a track to an online service which introduces the         song to a group of 1000 listeners online. These listeners are         also wearable computing device of the present disclosure users.         The service grants them access to music before it's released         publicly, on the condition that they use their devices to listen         to the music and agree to have their responses recorded for         marketing and research purposes.     -   Those 1000 people listen to the song in a neutral environment         when they're in a self-assessed neutral mood, and the effects it         has on their emotions, Brain State, and physical well-being is         recorded via the system of the present disclosure.     -   The emotional affect data is summarized and analyzed to create         an emotional profile for the music.     -   This emotional profile defines the song's genre.     -   Blingo is satisfied after his busy day, and pops on a playlist         at home. He sets the playlist to sort by genre, where he chooses         “Wind-down”—the songs on this playlist have all been shown to         have a calming effect on the human mind.     -   “Emoshift”—Emotional Shifting of Piece of Music     -   Definition: A rapid shift from one emotional state to another.         Emoshift can be accomplished by listening to music. We try to         create an emoshift or other times we want to catch emoshifts.         Have meta-awareness of the shift and therefore transform         undesirable emotions into positive attitude.     -   Artist—can track emoshifts in users to get feedback on the         creative process.     -   User—Wants to change their mood and can use music to archive the         emoshift.     -   Advertisers want to understand how to use emoshift to influence         their audience. Advertisers can identify users that respond to         the music they include.     -   Emoshift Target—Specific goal of changing mood. Key         change—elevating tenor—famous key changes—moment song shifts key         change.

The value proposition for this user story may include: This service would allow listeners to help define the impact of music on their Brain States, and help them learn more about the sounds and pieces of music that have beneficial (or negative) effects on them as individuals.

Sensors used may include: Detecting emotional state of diverse users listening to a single song, and transmitting that data remotely. Analysis of that data to produce an “emotional effect” that it has on people.

Applications may include: Promotion, marketing, market research.

Example Application: Social Networking Through Musical Preferences: Choose Your Own Adventure Pop Music Phrases

This example application involves choose your own adventure.

The Following is an Illustrative user Story for this Example application:

-   -   Pearl is a pop music producer.     -   Her craft is to create punchy, catchy songs which capture the         hearts of millions.     -   In order to increase engagement with her pop songs, she decides         to leverage the fact that many people listen to music while         wearing brainwave sensors in their wearable computing devices of         the present disclosure.     -   The artist Pearl is working with has a clear dichotomy in his         fan base. Some take him really seriously and love to hear his         soulful side, while others just see him as a party animal.     -   Pearl decides to build on this by creating two different endings         to his latest song.     -   When listener brainwave data is available, the song analyzes         that data and determines the listener's response to the song.     -   When the listener is excited and engaged by the rhythm of the         music, the song automatically plays the upbeat dance-party         ending to the song.     -   When the listener seems more verbally engaged, listening to the         lyrics and connecting with the songwriter, the more soulful         ballad-esque ending plays.

The Value Proposition for this user Story may Include:

-   -   This technology would allow listeners to “emotionally re-mix”         the song that they are listening to. Depending on how they're         feeling, a song might turn out a different way, or take on a         different key. They can choose to “fade out” a song, or end it         with a bang. This provides a soundtrack to everyday life. This         technology would also allow music listeners to learn more about         the emotional effects of music—music teachers could use it in         the classroom or in a tutoring context to help students learn         about how they and their classmates respond to music on an         emotional level, all while experimenting with that music.

Sensors used may include: Brainwave sensors, motion detectors, gyroscopes.

Applications may include: Social networking, apps, music composition, music education.

Example Application: Social Networking Through Musical Preferences: Social Networking Appreciation of Music

This example application involves a Speakeasy with Music as Gathering Space/Band Together.

The Following is an Illustrative user Story for this Example Application:

-   -   Billus loves music, and has a vast collection of old and rare         music recordings from artists many of his peers haven't heard         of.     -   Billus listens to his music on a random shuffle setting, and         when a particular track comes on, his wearable computing device         of the present disclosure detects his nostalgia and deep         emotional reaction using a future function of the device, while         simultaneously recognizing the song through its built-in         microphone.     -   The nature of the emotional moment Billus has sends him access         to a special web forum around that particular song.     -   This entry only happens when a person brainwaves are classified         into a specific type of emotion and level of emotion. The         curator of the group decides on the level and type of emotion         she/he is willing to join this group.     -   When he accesses that web portal, he's able to see lively         discussions and other content generated by fellow music lovers         who all love that same song/artist/album/etc.     -   The artist of that song is much older now, but has provided some         special content on the page, which is only accessible here.     -   Every single person on that web forum by definition has had a         similar genuine emotional response to the same piece of music,         that's how it allowed them all to access the forum.     -   The forum is off limits if you have not had this response.

The Value Proposition for this user Story may Include:

-   -   People have always gathered around music. Whether it's spiritual         or social, music creates a gathering space. Most major social         movements also have a musical aesthetic associated with them—for         example, the Jazz Age is inextricably linked with innovations in         the art of popular music. At all points in history, people have         banded together through music.     -   Consider the moment in Garden State when Natalie Portman's         character introduces herself to the film's protagonist by         putting a pair of headphones on his head and sharing her         favourite song by The Shins with him. This technology would help         facilitate that interaction.

FIG. 7 shows an exemplary system implementation of the present disclosure for implementing this user story. In this architecture, the EEG signals of the user are classified while listening to a specific audio track of an artist. The audio has a unique identifier. There is a Private Web Site that is run by a moderator. The moderator decides which brain states and the strength of the brain state to allow new users into a private forum where people can share their feelings and thoughts about the audio track.

Example Application: Social Networking Through Musical Preferences: Asynchronous Merging of All Listener Data

This example application involves asynchronous merging of all listener data.

The Following is an Illustrative user Story for this Example Application:

-   -   Johnny logs on to a new website which offers him a chance to         listen to his music in new ways.     -   He finds an interface which asks him to input a song he wants to         explore.     -   Johnny decides to pick his favourite song, “Believe” by Justin         Bieber.     -   Johnny is confronted with metrics for the song based on people's         neural responses to the song over time, generated from aggregate         data on contextual baseline Brain State established by wearable         computing devices of the present disclosure.     -   The metrics can give him a different sense of what is enjoyable         about the song, and how other people respond to it.     -   He finds other people who started dancing in their chairs at the         exact second that he did while listening to the song, and         realizes he's not so ridiculous for liking Justin Bieber (or any         major pop artist).

The Value Proposition for this user Story may Include:

-   -   This service would allow listeners to learn whether their         response to a song is distinctive, or whether they share that         same reaction with other listeners at the Brain State level.         This helps concretize the deeper connection that some people         feel to specific songs.

Sensors used may include: Brainwave sensors (bone conduction).

Applications may include: Marketing, promotion, music engineering, sound engineering, social networking.

Example Application: Social Networking Through Musical Preferences: Facebook My Musical Emotion (L1)

This example application involves Facebook (or other social network platform) with a My Musical Emotion (L1).

The Following is an Illustrative user Story for this Example Application:

-   -   D'Angelo is feeling sad because he failed his driving test.     -   He is also wearing a wearable computing device of the present         disclosure.     -   The system of the present disclosure uses available sensors to         detect a change his mood from contextual baseline.     -   Based on his individual user settings, the device offers to play         song to match his perceived mood.     -   D'Angelo agrees, song starts to play     -   The system updates his social media profiles with news about his         mood and his song choice.

The Value Proposition for this user Story may Include:

-   -   This technology adds layers of meaning and connection to         existing social media content, based primarily on musical taste         and mood detection.

Sensors used may include: Bone conduction.

Applications may include: Music retailing, social media, listening to music at home; at work.

Example Application: Social Networking Through Musical Preferences: Emotional tagging (L1)

This example application involves emotional tagging (L1) or content enhancing.

The Following is an Illustrative user Story for this Example Application:

-   -   Bree is listening to a song on her mobile device.     -   She has technology of the system of the present disclosure         attached to that mobile device, either via an app or a separate         wearable device.     -   The system scans Bree as she listens to her music.     -   Differentiating changes in her Brain State, indicates Bree is         very relaxed.     -   According to her individual user privacy settings and with her         permission, Bree's mobile device posts the song's title on         Bree's Facebook or other social network wall, along with         emotional state (Bree is relaxed right now).

The Value Proposition for this user Story may Include:

-   -   Music and emotions are closely linked. People may want to         purchase music that corresponds to their emotional states.

Sensors used may include: Bone conduction, eye-tracking, heart-rate.

Applications may include: Social media sharing, marketing.

Example Application: Social Networking Through Musical Preferences: Audience Measurement (L1)

This example application involves audience measurement.

The Following is an Illustrative user Story for this Example Application:

-   -   The Shondells have a new album that's set to come out soon. They         want to have a party to celebrate. The audience at the party is         equipped with brainwave sensing technology of the system of the         present disclosure.

Changes in contextual baseline brain state are recorded during the performance. Changes in contextual baseline brain state can also be fed back to to Shondells in real-time illustrating audiences relative involvement in their musical pieces. Shondells adapt their musical creation in real-time to connect in a desired way with audience members.

Marketing of the album is tailored to the emotional response and aggregated data.

The Value Proposition for this user Story may Include:

-   -   A new kind of “focus grouping” based on brain scan metrics. This         would be a standardized metric to a person's emotional response         to music, entertainment, etc. People's internal mood and thought         state is next to impossible to interpret. A new kind of way to         “intuit” the moods and state of individuals in real-time can         provide information to adapt our response and approach in         real-time based on their personal data.

Sensors used may include: Bone conduction, galvanic skin response, temperature, gyroscope, accelerometer.

Applications may include: Marketing movies, music, TV shows, etc.

Example Application: Social Networking Through Musical Preferences: Sound Collage (L1)

This example application involves a sound collage.

The Following is an Illustrative user Story for this Example Application:

-   -   Jackson wears his wearable computing device of the present         disclosure on a regular basis, most of the day almost every day.         It monitors his brainwaves and every sound he hears (although it         does not record them).     -   Over time it learns to correlate his emotions to particular         sounds that he hears.     -   Later, when Jackson is really frustrated, he wants to share it         on his favorite social media site.     -   He types a status update into the social media app and presses a         button on his phone.     -   The system checks the last few minutes of his brainwave         recording for any significant emotional events.     -   The system identifies frustration as the most prominent emotion.     -   The system then combs through the audio recordings of Jackson's         life, and looks for sounds that are correlated with similar         levels of frustration.     -   The system assembles a sound collage of frustrating sounds.     -   When Jackson's friends click on the status update in their         newsfeed, they hear the sound collage.

The Value Proposition for this user Story may Include:

-   -   This functionality correlates specific sounds with specific         emotions. The system of the present disclosure allows the         detection of this relationship between stimuli and response. It         also encourages users to create sound “collages” or “mixes” of         sounds that make them feel a certain way.

Sensors used may include: microphones, brain sensors.

Applications may include: social media.

Example Application: Social Networking Through Musical Preferences: Emotional Data-Mining of Public Spaces

The Following is an Illustrative user Story for this Example Application:

-   -   Trent is designing an opera house and performance space.     -   Trent wants to know how visitors to his space will feel in it.     -   Working with a design research team, he accumulates data on 100         people visiting performance spaces similar to the one he intends         to build.     -   The people in this sample are all wearing wearable computing         devices of the present disclosure when they attend concerts,         lectures, films, or other performances.     -   The system of the present disclosure logs specific changes to         their Brain State and begins recording or “listening” when those         changes occur.     -   Eventually, the system generates aggregate data on sound quality         within a performance space, and how visitors react to sound         quality.     -   For example, some users might feel mildly annoyed at poor sound         quality, but not feel compelled to leave the space. Others might         not like how far away from the stage they are, but appreciate         the quality of the sound.

The Value Proposition for this user Story may Include:

-   -   For example, this may be central to what the system is capable         of doing. What the system does is harvest Brain State data in         context. It registers changes in Brain State and interprets the         wearer's emotions based on those changes to contextual baseline.     -   Learn how people feel about the sound of a space. Apply these         learnings to a building, space, game etc.

Applications may include: Architecture and urban design, design of spaces dependent on sounds like sports stadiums and major performance venues.

Example Application: Social Networking Through Musical Preferences: Emotional Arena

An emotional arena is a physical or virtual space that is engineered to promote a certain emotional state among people in the arena. Emotional Arena can be for entertainment, but it's also an opportunity to create an experience unlike any other previously available.

The Following is an Illustrative user Story for this Example Application:

-   -   Frances is an architect. She is creating a new shopping mall.     -   Frances wants to know how sound will influence shoppers.     -   Working with sample shoppers in malls she wants to emulate,         Frances asks her sample to wear the wearable computing device of         the present disclosure while going shopping.     -   Frances is interested in how shoppers listen to or ignore         announcements over loudspeakers, and how she can speed up crowds         of people using music or sound.     -   The statistical sample wears the wearable computing device while         they go shopping.     -   The system of the present disclosure takes note of their         emotional reactions to auditory stimuli.     -   The system then maps these data points against location and         sales data within the store. (Where was the shopper? What was on         sale? How many people were around them? Had they just heard an         announcement?)     -   The system generates data on these reactions and sends it to         Frances.     -   Eventually, Frances is able to design environments that allow         for a free flow of people, but still facilitate retail sales.

Sensors used may include: Cameras, microphones, bone conduction (EEG).

Applications may include: Theme park design, experience design, experience prototyping, architecture, urban planning and design, museum and exhibit design, outdoor design.

Example Application: Improve Performance: Dance Games with Head Movements (L1)

This example application involves dance games with head movements (L1).

The Following is an Illustrative user Story for this Example Application:

-   -   Roc is the master of games like Dance Dance Revolution, and         other games that require the player to dance based on visual and         audio prompts. Roc is very competitive with these games and         participates in online and arcade guilds.     -   He wants to improve his skills at these games, so he buys a         wearable computing device of the present disclosure.     -   The system of the present disclosure has a strategic partnership         with game developers to add outputs to the system to enhance         gameplay and make it more interactive.     -   The game incorporates head movements he makes during gameplay.         These movements come from the accelerometer in the headband of         the wearable computing device.     -   He becomes an even better dancer, and improves his ranking in         the gaming guilds he's a member of.     -   Eventually, Roc gains entry to a prestigious dancing school         based on how much practise he's had with his devices as a         competitive gamer.

The Value Proposition for this user Story may Include:

-   -   This functionality would allow gamers to have improved gameplay         with music- and dance-based games, with more interactive fun and         enhanced participation.

Sensors used may include: Accelerometer; gyroscope; microphone.

Applications may include: Gaming, dance education, dance and music therapy, physical therapy, cardio exercise.

Example Application: Caregiver Interaction:L.1 Autistics and Alzheimers can hear other's emotions encoded as music (L1)

User stories included: 3.6.

The Following is an Illustrative user Story for this Example Application:

-   -   Al has an autism spectrum disorder. This disorder makes it         difficult to interpret emotions in others based on visual         information like affect, and difficult to express emotion in a         way that others who are neurotypical can understand easily.     -   However, Al responds to music and can infer emotion from it.         Music is one of his ways of communicating and sharing feelings.         It was how he first learned about the various nuances of emotion         and affect.     -   A loved one or care-giver of Al's wears a wearable computing         device of the present disclosure. The system of the present         disclosure detects salient emotion being expressed by its wearer         by analyzing and classifying their brain waves into brain state.         Brain State responses that are greater than contextual baseline         are used to drive musical phrases known to be associated with         emotion.     -   While wearing the wearable computing device, Al can “compose”         his emotions by creating complex musical phrases based on his         feelings. This is easier for him than communicating verbally or         in writing.

The Value Proposition for this user Story may Include:

-   -   This allows people with an emotion understanding deficit to         improve their understanding of another's emotions. It can make         them more responsive and improve their relationships with other         people. This method can also be used to help people understand         their own emotions through music and can have therapeutic value.         The same set-up can be used with Alzheimer's or other patients         that have difficulty interpreting emotions in others.

Sensors used may include: brainwave sensors

Applications may include: Home care of patients, therapy, training.

FIG. 8 shows an architecture drawing of the system of the present disclosure for an implementation of this user story.

Example Application: Stimulation Technologies

The Following is an Illustrative user Story for this Example Application:

-   -   A user is wearing an EEG headband to analyze their brainwave         characteristics while the user does a mental and or physical         exercise. The exercise may have environmental stimuli such as         background sounds, lighting, music etc. or lack of stimuli such         as total darkness, or silence. A user's brainwaves and other         physiological signals are analyzed and interpreted to provide a         characterization of the user. An example is the amplitude of the         peak power and frequency emitted by the user in the alpha range         (8 to 13 Hz). An external stimulation device such as         Transcranial magnetic stimulation

The Value Proposition for this user Story may Include:

-   -   can be used to enhance the user's ability to produce the alpha         peak and therefore enhance their ability to enjoy sensory input         such as music.

Additional Intelligent Music features

Another illustrative example User Story is Mindfulness of Music and Sound.

A user is played sound/music during a session. This could be: a. random snippets of sound at random times (birds chirping, someone making dinner, a protest march, classical music, etc); b. continuous sound with predictable changes; c. dynamically adjusting depending on the state of mind of the user: increasing difficulty as the user has more focus and less difficult if the user is distracted; or d. user has the ability to choose their difficulty level, for example.

The EEG monitor tracks their state of distraction during the session.

The user has the ability to turn on feedback of their distracted state. The user gets a score at the end of the session.

A distracted state, for example, may be thinking of the future(predicting what is going to happen) or remembering the past, or attention drawn to physical sensations (rather than the anchor of sound), drowsiness (laxity/dullness) or having any feeling except equanimity, joy, loving-kindness, or compassion.

In an aspect, embodiments described provide a system and method for music recommendation and effects using biological data.

A system and method may characterize music based on digital signal processing of epochs of a song to describe parameters related to human perception. Examples parameters or attributes include brightness, bandwidth, tempo, volume, rhythm, and so on. Data may also describe how these features change over time. These may be called sonic parameters of a song. An epoch is a length of time in how a piece of music is divided. An epoch may be short enough to capture variety, such as for example 200 ms. A user's music preference of songs may be calculated based on their choices of preference when comparing like/dislike across numerous pairs of dissimilar songs.

A human response to music can be characterized by a variety of means; examples among these are behavioural measures such as rhythm entrainment as measured by movement of one of more parts of the body, physiological changes in breathing, heart rate and heart rate variability (HR and HRV), muscle activity (EMG) or galvanic skin response (GSR), and especially changes in brain activity as measured by continuous or epoch-based electroencephalography (EEG), hemispheric asymmetry, and event-related potentiometry (ERP) corresponding to discrete or repeated events in the music or the acoustic environment. The brain responses in particular often correspond to specific listener experiences relating to emotional or arousal/engagement state.

Examples of continuous EEG measurements which reflect perceptual, emotional, or arousal/engagement to music are, but may not be limited to, spectral band power including relative contributions of delta, theta, alpha, beta, and gamma waves. They may also include synchronization or desynchronization (ERS or ERD)

One example of EEG measurement useful in distinguishing both the emotional valence (happy/sad) and the arousal (intensity) of music is alpha power.

When an element of music or the auditory environment changes, for example by increasing or decreasing in amplitude, or changing in frequency or timbre, these changes can be detected as transient changes in the electrical potential of the brain. Some of these changes in measured brain potentials are sensitive to entrainment or musical experience, wherein repeated exposure can enhance the strength of the signal detected by EEG.

Another form of evoked potential, the auditory steady-state response, has amplitude or phase components related to the listener's level of attention to the music or sound. Another potential may be referred to as the auditory mismatch negativity (MMN) reflects an unexpected change in a pattern of rhythmic, repeated sound stimuli, and can be detected in EEG using electrodes near the mastoid process behind the ear. Similarly, the early right anterior negativity (ERAN), which reflects the interaction of a music stimulus and the listener's memory, can be measured with a few electrodes located in the temporal and frontal areas of the scalp.

Embodiments described herein may provide music processing hardware (e.g. DSP) that extracts a set of sonic features from music data. The music data may include multiple songs over one or more time periods. The music data may be defined temporally to map to sensor data captured at different times. For example, a song is divided into N time epochs. A set of sonic features is extracted per epoch where V1 is the vector of sonic features per epoch 1, to VN vector of sonic features for epoch N. In addition, meta data for each song is provided that describes the artist, genre, release date, etc. A set of features can be calculated for all music and this set is known as Music-Everywhere-Everything (MEE).

Embodiments described herein provide music processing hardware that adds features extracted from biological data (from bio-signal sensors) about users (e.g. at playback of music) per epoch of music, temporal characteristics of user's song choices in portable music player devices (e.g. iPods), a user's personality or user attributes, and expert input. The biological data may be used to compute the user's level of engagement, valence and arousal with the music data on an epoch per epoch basis. This may provide a richer way of characterizing a user's response to music and build up models that may help predict more accurately which music selection a user will prefer. Also this process can be used to data-mine across a large population of users their biological reaction to music pieces. The distribution of biological reactions to a specific piece of music may be highly correlated.

Users whose biological reaction differ from the norm may be treated separately and in their own cluster as to what music selections they will prefer. In addition, a system and method are described where effects can be added to existing music to help a user achieve a target state (FIG. 15).

Embodiments described herein provide music processing hardware devices, systems, and methods that add temporal history to the selection of songs selected by the user. The temporal history of a series of songs listened to by a user are described using the following notation.

As an illustrative example, Sa is song A.

Further, Sa(t,i,j) may indicate that a user listened to song a at date and time t starting at ith percentage fraction of the song and ending at the jth fraction. Example Sa(Mar 15-2014-2:01, 0, 1) means that the user started listening to the song Mar 15-2014-2:01 and they started at the beginning “0” and listened to the song to its completion “1”

T(Sa(t,i,j), Sb(t,i,j)) may be the transition from listening to song a to song b.

Further, associated with Sa(t,i,j) is a set of features that describe the sonic properties of the song, meta data of the song, and the user's reaction to that song based on measuring their physiological response on an epoch by epoch basis.

As an example, features of Sa(Mar 15-2014-2:01, 0, 1) may be:

V1, V2, VN—set of sonic features per epoch of song.

B1, B2, BN—set of biological features of the user's reaction to a song using the same epochs for the sonic features. The set of biological features can include those extracted from EEG, EMG, ECG, GSR, as an example. In addition, the accelerometer features that describe the motion of the user are also extracted.

M1,M2 . . . Mm—set of meta data for the song. For example, Artist, year released, name of song, genre.

P1, P2, Pp—set of user profile characteristics that can include: birth year, birth city, genre preferences, gender, and so on.

U1, U2, . . . Uq—set of user actions taken during the playing of this instance of the song. Examples are turned the volume up. Adjusted the equalizer such as bass and treble settings. Other features are that this song was specifically selected by the user or this was just the next song in the playlist. Sometimes a user sets their music player to shuffle songs so an action could be that this song was skipped after listening to first 5 seconds. Another set of actions is that this song is part of a user created playlist. People can shuffle songs in the context of artists or genre or release year, and so on.

C1, C2, . . . CN—set of labels or classes associated with an epoch of music. This is the output of the predictive model. These may be used in supervised machine learning to create the predictive model.

Precise Universal Time-Stamps

Embodiments described herein provide music processing hardware that provide precise universal time-stamps in music data to map to time-stamps biological data and other data sets.

Music requires precise time-stamps. With digital technology, tracks may be clipped. It is important when tagging a piece of music with emotion (via bio-signal data) that it corresponds with the music event data (e.g. note) that evoked that emotion. This may enable a large number (e.g. thousands of user's responses) to be synchronized to the same musical event that evoked the emotion. In this way, machine learning and statistical analysis may be applied to precisely the same moments in music data across a large population users (as expressed via bio-signal data). Music tracks may be standardized as to when a track starts so that time-stamps are universal across all presentations of a specific track of music. A specific note that is characteristic of the song may be tagged with a time-stamp as the start of the song, i.e. START-NOTE.

A user may have a continuous or never ending history of songs that have been listened to. The history of songs may be defined as music data. One example sequence is: . . . Sa(Mar 15-2014-2:01, 0, 1); Sd(Mar 15-2014-2:10, 0.2, 0.9); Sb(Mar 15-2014-2:14, 0, 0.8); . . .

EEG can be used to infer a user's emotional response to music data using event related synchronization or desynchronization, event related potentials, asymmetry across hemispheres, coherence across a set of electrodes, and power in specific frequency bands. EMG can be used to determine a user's level of muscular tension, their movement in time. ECG can be used to infer the level of a user's arousal. These biological features can be correlated to the sonic features of the music using universal timestamp mapping, for example.

Different sensors may capture and provide different types of bio-signal sensor data. For example, an accelerometer and gyroscope may be used to infer if the user is moving synchronized to the music (e.g. dancing). Accelerometer and Gyroscope can be used to classify rhythm in music and match to user's movement, or detect any rhythm in user's movement, for example. If a user is moving in rhythm to the music then this can be used as an input to classify the level of engagement of a user with the music, and the likelihood of entrainment and engagement with a particular rhythm which may in turn indicate the likelihood of a user's preference for other songs or musical excerpts with similar rhythms.

Category of an Epoch (C1, C2, CN)

An epoch can be labelled with a set of one or more categories that the epoch belongs. Categories may be nested in a hierarchy or an ordered list since a number of states can co-exist in the human body simultaneously. Categories may be expressed as a probability of the class or nominal label. Example categories may be as follows:

Preference of music: like/dislike etc. or valence.

Emotions: Positive or negative states someone is not necessarily directly aware of.

Affect: When one becomes aware of their emotions, or expresses them overtly (such as smiling or frowning).

Mood: A general, overall emotional state based longer-term changes in emotion. Can be inferred from regular reports of affect.

Physiological states: sleep stages (awake, drowsy, deep sleep, REM), arousal.

Cognitive States: mental effort, task engagement, frustration, focus.

Motion and muscular contraction, Tension, Walking, Running, Sitting

All of the features may be stored as data structure in a cloud based storage implementation, as an example. One example is that people are wearing sensors such as brain sensing headbands, and other biological sensors that are connected to a mobile device (e.g. using bluetooth, NFC, wireless). The device plays music and collects the biological sensor data. All of the streams of data flowing through the music player/biological sensor integration device tag the streams with time-stamps so that all of the data that occurred simultaneously are tagged with the same or a corresponding time stamp. The device can upload the data (music, biological sensor data, user actions, music meta data, etc.) which can be uploaded to a Cloud for storage and processing. Please refer to Applicant's U.S. application Ser. No. 14/115,781 entitled Systems and Methods for Collecting, Analyzing and Sharing Bio-Signal Data and Non-Bio-Signal Data, the entirety of which is incorporated by reference herein, as reference for how cloud storage may be used to process the data, create predictive models and create analysis algorithms that are used to interact with functionality described herein.

One example of determining a predictive model is the Hierarchical Temporal Memory (HTM). HTM can infer the next set of features that will have high likelihood or probability to create an emotional response in a user. A song is recommended to the user. The user can accept, or modify (i.e. reject, or choose another song) the system's music recommendation. This may be referred to as MANUAL OVERRIDE or feedback. The user's choice can be used as additional information about the song (e.g. metadata) and the user's preference of that song.

The predictive models of embodiments described herein may have a learning mode and an operational mode. The learning mode may be where the model is being updated as new information arrives based on user choices and biological reactions to songs. Sometimes the choice of next song that is offered may be random and based on the biological reaction of the user can be used to determine how anomalous the system's recommendation. Random choices and/or songs that are not part of the usual space of choices may be useful to expand the accuracy of the model by probing new genres that may be of interest to the user.

Operation mode occurs when the model offers a recommendation of a song to the user.

Smart Playlist Controller

In an aspect, there is provided a music processor that provides a smart playlist controller. The smart playlist controller may generate queries or questions to receive additional information about a user.

Example questions for User to help select music may include:

-   1. How do you feel right now? -   2. How do you currently feel? How is the current music making you     feel? or -   3. What is your goal emotion? or How do you want to feel? or What     kind of music are you looking for? If you are sad and want to hear     sad music then user is looking for empathy. If the user is sad and     want to happy then they want to improve their mood. -   4. Based on answers to questions 1-3, select appropriate music.     (PROCESS, OUTPUT)

The smart playlist controller may automate selection of music for the user that will help them meet their goal emotion according to some embodiments. The may help automate questions 1 and 2. User input may be required for question 3. Step 4 may generate output of embodiments described herein.

The user can select a target state that they want to achieve, which, as an example, may be on two axis. One axis may be a level of energy and the other access may be a level of engagement or attention they want to invest in the music. Sometimes people may want unobtrusive background music and other times they want to engage 100% of their attention in the song. In addition to the state that the user wishes to achieve, their current biological state as inferred through analysis of their biological sensor data may be displayed on the same two axis.

FIG. 10 shows an example of a user interface for describing the current user state and target state, which may display an Energy—Engagement matrix which may be an interactive space for user input.

In another example of target state, the user may choose along Valence and Arousal (VA) by a selection on an input device (e.g. touchscreen display). In these dimensions, Valence may be on one axis with “approach motivation” (feelings of positivity) on one end, and “avoidance motivation” (feelings of negativity) on the other. The other opposing axis may be Arousal, with high intensity of feeling on one side and low intensity on the other. Commonly-felt emotions traditionally fall within the quadrants formed by that VA matrix. Again the user's current state (i.e. the answer to question 2. above) may be displayed on the VA matrix and they press on the VA matrix where they the music to take them. See FIG. 11 for an example of this user interface with a valence arousal interactive space for user input.

As an illustrative example, the user can select that they want more energy for interacting with a section of the matrix shown in an interactive display, for example. The difference between the user's target state and their current state may be represented as a vector. This vector can be used to select or recommend songs that may help the user achieve their target state. The user for instance can express the desire to be happier as a target state. The difference between their current biological state and their target state (i.e. as represented by a data structure vector) can be used to select the attributes of the music to offer. If there is a large difference in happiness from a user's current state and target state then songs with greater scores for happiness may be offered to the user.

Another way for the user to express their desired target state may be by selecting a word from a list of words offered to the user in a display interface. The list may include selections such as happier, energetic, relaxed, peaceful to represent a desired target state. These can be isolated into a section of the Valence-Arousal (VA) quadrant. As described, the user's current emotional state may be located in the VA plane. A vector can be found from the user's current state to their target to drive the type of music that they wish to hear.

How to Determine the Emotional Tone of a Piece of Music

FIG. 12 shows an example architecture to implement a predictive model engine according to example embodiments. The model engine may predict human reaction of music.

In general, there may be strong agreement across many people as to the emotional tone of a piece of music. Most people may agree that a piece is happy, sad, romantic, energising, and so on. The association of a song to its emotional tone can be learned across many people (and bio-signal data associated with the many people) inferred from their biological features or it can be part of the meta data of the song, for example. Italian musical terms of classical music is an example of meta data describing the emotional tone of a piece of music such as con brio (with vigour), dolce (sweet), dolente (sad), dolore (grief) etc. Using biological features along with their corresponding sonic features of a piece, embodiments described herein may be configured for machine learning of the biological signatures per epoch associated with the sonic features of the epoch of music. Music is an emotional language and composers use its vocabulary to create emotional tone for their piece. A sad song may have a lot of minor and diminished chords to it, for example. A happy song may be set in major keys. Angry music may have a driving beat and minor keys. A key is a pattern of notes that makes up a scale. The keys of music may be defined in music data. Different keys have different patterns of notes separated by different intervals. It is the different intervals between notes that distinguish major keys from minor keys. In addition tempo (rate of speed of music) can affect the emotional tone of music, for instance slow tempo make music seem sad. Passages of music with known emotional tone can be used as labelled training data for supervised machine learning. The emotional tone of music of part of the meta-data of the music piece may be represented as M1, M2, . . . Mm. Music processing hardware may process music data to extract meta-data.

Detection of Approach and Avoidance may be implemented by some embodiments, for example. Alpha asymmetry across brain hemispheres may be a measure of approach and avoidance. Heart rate goes up and down with arousal. Heart rate goes up both with a response of excitement and also an aversive response. Changes in body tension can also be a sign of aversion but also excitement with anticipation of a release.

Two methods are described herein of creating a model to predict the emotional impact of music on a user as illustrative examples. The first example method is a two-stage method. First, the labelled epochs of music may be used to train the system with the sonic features of those passages. Example types of supervised machine learning include support vector machines, deep learning neural networks, decision trees, logistic regression, linear discriminant analysis and others. In addition, temporal information can also be encoded using Hidden Markov Models. Using a supervised learning method a model can be trained by system based on the known examples of labelled epochs of music. For unknown pieces of music the model generated and implemented by the system or music processing hardware can be used to predict the emotional tone of a piece of music based its sonic parameters represented as V1, V2, . . . VN. The model for predicting the emotional tone of music based on its sonic parameters can be used to classify all music of interest. In this way a small set of human labelled music can be used to classify, using a configured machine, a large catalog of music data. The second stage is to learn the biological or bio-signals evoked when a user listens to a piece of music and their extracted biological features B1, B2, . . . BN. The model is built the same that results in a predictive model of classifying a user's biological signals into an emotional state.

The second example method uses unsupervised machine learning methods such as HTM or deep learning where training the combined features of both biological and sonic parameters simultaneously. One example is the Hierarchical Temporal Memory (HTM). The deep learning method works by using several layers of stacked two-stage (hidden, visible) restricted Boltzmann machine. Deep learning can discover the model of the data using unsupervised methods. Deep learning may be used for optical character recognition and speech recognition, for example. In addition adding Hidden Markov Models to the output of deep learning can improve the accuracy prediction by bringing in temporal characteristics of the music. All of the features of both the sonic parameters of the music and biological parameters can be fed to the model. If there is a sufficient number of training samples, the raw itself (notes, voltage values of samples biological data, and so on) can be used to train the deep learning networks. Once a deep learning process implemented by the system or music processing hardware has learned the data either through its features or its raw data, the data can be provided as labelled examples to turn the deep learning network into a predictive model. Then the network and the weights can be optimized by the system or music processing hardware for further refinement.

Another example of improving the accuracy of the categorization of a piece of music is to use Hidden Markov Models. As explained above happy music may be written in a major key while sad music may be written in a minor key. Numerous examples of music with major keys and minor keys can be used to train two different Markov Models. The Markov model may be expressed as a set of states and their transition probabilities. A state in this case may be a set of features (e.g. representative of a note and the tempo of change from a preceding note). All examples of music in a key can build a model that captures the probability of sequence of notes and tempo from a starting note. So one model is built for major key Model1 and another model is built for minor key Model2, as an illustrative example. The sequence of a set of notes can be described from their vector of features V1,V2, . . . Vi. An unknown piece of music has two aspects which may be calculated:

p(V1,V2, . . . Vi|Model1) * p(Model1)—conditional probability of feature vector given model 1 times probability model 1

p(V1,V2, . . . Vi|Model2) * p(Model2)—conditional probability of feature vector given model 2 times probability model 2

An unknown piece of music can be classified as belonging to Model 1 (major key) or Model 2 (minor key) by choosing which of the above equations has the higher value. This piece of information can be used to revise the sonic parameters of a piece of music by including this label (sad or happy) into each epoch that was used to determine the key.

The process described herein can also be used to improve the classification accuracy of the biological state of a person. In this case

p(B1,B2, . . . Bi|Model1) * p(Model1)—conditional probability of feature vector given model 1 times probability model 1.

p(B1,B2, . . . Bi|Model2) * p(Model2)—conditional probability of feature vector given model 2 times probability model 2.

Like and dislike of music pieces by an individual is a separate dimension than the emotional tone of music. The emotional tone of music is an important determinant of whether a user will like a piece of music but other factors are present as well. The like and dislike of music can be partially determined from the user's action when selecting music, i.e. features U1, U2, . . . Uu. For instance increasing a piece's volume, whether a song is skipped, the frequency of playing a song are examples of features that are associated with like and dislike of music.

These models can be trained across a population of users whose biological signals have been recorded while listening to music. These general models can be fine-tuned to specific users. One way is to use manual over-ride or user feedback which is described herein. If sufficient data exists for a user then a customized model can be trained for them using the methods described herein. Another method is to match this user to others using meta data about the person. For instance age, genre of music, favourite artists, etc. This can help localize within the universe of users that have similarity to this user and using that data to develop a model for that sub-group of users.

Manual Override or Feedback

The machine learning methods described herein can be fine-tuned for each user. The biological signals of a user while listening to music may be used to train a model for that user. The user may choose to ignore the choices offered by the system. This manual override is also input to the system to help it learn the user's preferences. The user may revise their vector to emphasize that the choices suggested by the system are not happy enough. In addition, the user's preferences can be used to develop a model of like/dislike of music and that person's preferences.

EEG and the other biological signals allow us to estimate the user's current state of engagement with the music, valence and arousal. FIG. 14 is an example of 9 different users, of several sessions where users undergo a memory recall exercise of categories. These are unique signatures of each individual that tend to cluster tightly compared to a large population of other users. This information can be combined with the time of day to capture the patterns of change that happen with varying circadian rhythms and sleep pressure. The location of an individual's cluster can be compared to the whole. Models can be customized to an individual and as well as adapt over time by comparing the statistics of features related to the an individual's cluster to the population as a whole. Also models can be specific for sub-groups based on age, gender, handedness. These covariates are known to be associated with certain biomarkers of EEG information.

Another example is clustering of sessions of individuals. In the PCA space, sessions of the same users tend to cluster together. Each plot represents the PCA cluster of an individual. All the sessions of a 9 users were chosen at random from our cloud data. The plot reveals that a user's relative EEG power tends to cluster in a local space. This can be used as a biometric marker of that user. The lighter points each represents a single session. The larger dots are increased the size correspond to the sessions of a unique user.

This information adds another dimension to people that are brought together over the love a song. People whose clusters overlap may exhibit similar personalities or tastes.

FIG. 13 illustrates a temporal modelling of sequence of music choices, and shows features extracted from a sequence of songs provided as input as music data. An Example of notation of a sequence of songs may be

. . . Sa(Mar 15-2014-2:01, 0, 1); Sd(Mar 15-2014-2:10, 0.2, 0.9); Sb(Mar 15-2014-2:14, 0, 0.8); . . .

The system adds temporal information of the sequence of songs. The new set of data may be referred to as Music-Everywhere-Everything-Temporal (MEET).

These stream of songs Sa, Sd, Sb, . . . and their associated features including the user's biological response may fed into the system or music processing hardware as input and may be used for unsupervised learning of temporal models. One example is the Hierarchical Temporal Memory (HTM). HTM learns the structure of the data based on the temporal order of the features it is fed. HTM can be used to determine the novelty of an event of data that it is fed. For a given event k and based on the sequence of events prior event(k-1),event(k-2) etc., HTM can determine the likelihood of an event k. This model may be continuously updated by the system based on the user's behaviour as input by bio-signal data. HTM learns to understand a long chain of temporal events. The embodiments may not considers music pieces as sole or isolated examples in the training data and may consider the temporal relationships revealed by listening to music one selection after another.

Another example of temporal modelling is using Hidden Markov Models (HMM). According to some embodiments, an HMM can describe a network of transitions from one song to another showing the probability of transition from one song to another. For example, a user is listening to the Beatles song “Let It Be” in the album of the same name. By looking across user choices of next song it could be revealed that the next song to be selected is “Maggie Mae” with a high probability. This is because “Maggie Mae” is the next track in the album “Let It Be”. However, with digital music, a user is not constrained to hear tracks in this order and the order of songs is based on some goal or mood that the user wants to achieve. The order of songs selected can be useful in choosing songs that are tied together. The set of songs after the current song listened that have high probabilities of being selected form a cluster of songs. These clusters have a great deal of similarity and can also be labelled with meta data such as artist and or genre. The clusters can also be considered “genres” in themselves. These clusters based on high probability of being played after or before a song can also be called “genres” and can be added to the meta data of the song that can improve the machine learning models that are created.

FIG. 14 shows an example of clustering of sessions of individuals.

In the PCA space, sessions of the same users tend to cluster together. Each plot represents the PCA cluster of an individual. All the sessions of a 9 users were chosen at random from our Cloud data. The plot reveals that a user's relative EEG power tends to cluster in a local space. This can be used as a biometric marker of that user. The smaller light grey points each represents a single session. The darker larger dots are increased in size to correspond to the sessions of a unique user.

FIG. 14 shows the architecture of a Music Effect Controller to influence user state.

As an illustrative example, a Disc Jockey (DJ) is a person that mixes and adds effects to recorded music to a live audience. A human DJ can interpret an audience's reaction to help them improve the level of engagement and satisfaction that an audience receives from their performance. A relationship and communication form between the DJ and the audience. The DJ is doing a live artistic performance and they have many tools on hand that can affect the underlying recorded music that they are working with.

FIG. 14 describes a system where the user inputs a target state they want to achieve and a control system that changes the effects of recorded or live music. FIGS. 10 and 11 are example user interfaces in how a user can set their Target User State. Biological sensors measure the user's response to the music and features are extracted from these signals as shown in FIG. 12 (Biological Feature Extractor). Also the sonic parameters are extracted from the music as shown in FIG. 12 (Sonic Feature Extractor). The controller changes the effects its applies to the music data to be output by setting the parameters of the Music Processor. The controller uses feedback from the Biological Feature Extractor and the features of the music output from the Sonic Feature Extractor to change the parameters (music effect parameters) of the Music Processor to adapt to helping the user reach their desired Target State.

The Following is a List of Music Effects that the Music Processor can Apply:

-   -   change of tempo     -   filtering—changing the amplification level of different         frequency bands in the music     -   changes in left/right balance     -   reverberation     -   spatialization     -   room models     -   pitch shifting     -   chirp effects     -   vocoder (human voice as notes)     -   auto-tune     -   auto detune     -   humanize electronic music     -   the order of samples of music may be changed (i.e. the chorus         and verses may be ordered in different ways to change the order         the music is played but the piece is still recognizable.)     -   profile of tension and release over the course of a piece of         music

In the method described in FIG. 15, the function of the DJ (i.e. Controller) may be automated by system by applying a process that automates the DJ's interpretation of the user's reaction. The Controller changes an effect of the music such as increasing the volume. The user's reaction to the change is based on the prediction models of signals from their biological sensors. If a user has a positive reaction the change is kept. If the user has a negative reaction that effect is not added to the music. The rate and number of changes are parameters of the algorithm that can be adjusted based on the preferences of the user.

One method that the controller can use to estimate the Music Effect Parameters is by finding the maximum in the following condition probability:

P(ME|V, B, T) is the conditional probability of the music effects given the sonic features of the music, the user's initial biological state and the user's target state.

-   where -   ME is Music Effects Parameters -   V is the vector V1, V2, VN—set of sonic features per epoch of song -   B is the vector B1, B2, BN—set of biological features of the user's     current state prior to applying changes in effects -   T is the target user state that the user wants to achieve. See FIGS.     10 and 11 for example of how user can set their target state.

P(ME|V, B, T) can be estimated using machine learning techniques such as probabilistic neural networks, logistic regression, Fischer discriminant analysis and other techniques. The range of values of Music Effect parameters can be supplied by human music experts. Training examples can be obtained across hundreds or thousands of users. The model can be general to a population, to a sub-group (i.e. genre) or to an individual.

The Controller uses the features from the Biological Feature Extractor to determine the current User State. In one example the User State can be described by four parameters: a) Valence (positive or negative emotion), b) Arousal, c) level of attention, and d) level of synchronization. Estimating valence and arousal are described in prior art. Level of attention and Level of synchronization is described in more detail herein. See also, for example, Applicant's U.S. application Ser. No. 14/368,333 entitled ADAPTIVE BRAIN TRAINING COMPUTER SYSTEM AND METHOD for a description of how a busy-mind score is calculated, the entirety of which is incorporated by reference herein. Further control details may be found in Applicant's PCT Patent Application No. PCT/CA2013/000785, filed Sep. 16, 2013, the entirety of which is incorporated by reference herein. Further examples of modifying the presentment of digital content may be found at Applicant's PCT Patent Application No. PCT/CA2013/001009, filed Dec. 4, 2013 the entirety of which is incorporated by reference herein.

Music has a beat that can be considered a very strong stimulus to the user. Level of Synchronization can be described by four reactions user has to the beat in the music: one they may be anticipating the beat, be in synchrony with the beat, lagging or not following the beat. In neuroscience Event Related Potential (ERP) are signals seen in a user's brain signals in response to a stimulus. Typical delay for a human brain to process and create a response is 300 ms measured from the time of the stimulus and when the ERP is measured from the user's scalp. i.e. P300 ERP. A user is synchronized with the music when over the course of a minimum number of beats, the ERP associated with the stimulus is less than the typical delay in that response. For instance, the typical delay may be 300 ms but the user is creating ERPs within a range of plus or minus 100 ms relative the onset of each beat indicates the user is in sync with the beat. User is anticipating the beat when their ERP is precedes the stimulus by a minimum say 500 ms in this example. The user is lagging if the ERP is after the stimulus by a large amount say greater than 700 ms on average. The user is not following the beat if the distribution of the ERPs relative to the stimulus (i.e. beat) is random and has whose variance of the difference of the ERP to the stimulus is greater than a threshold and has a two tailed distribution (i.e. the ERP both lags and precedes the stimulus over a range of sequential beats). The Level of Synchronization is strongly related to the user's enjoyment and engagement with the music. If the user is in synch or anticipating the beat then the user is engaged in the music. If the user's synchronization is lagging or not following the beat then they are not enjoying the music or it is too fast for them to follow and it sounds like noise to them.

Another method of detecting rhythm entrainment and engagement is to use a combination of frontal alpha for engagement and emotional valence plus accelerometer data to detect entrained movement (even small movements). People seem to “prefer” a beat that they've already entrained to. We could, simultaneously analyze a musical rhythm, an EEG response, and EMG or accelerometer data to determine a) whether they're entraining to a beat through movement (toe-tapping or head-bobbing, for example) and b) whether their entrainment predicts the tempo and rhythm of the next song/excerpt to play that will generate a positive valence or sustained emotional state. This method can be used to help select the next song that would maintain the same beat or rhythm.

Probing a User to Improve Estimate of their Current Brain State

The system described in FIG. 15 can also be used to estimate the user's current state by using salient parts of a music piece or insert an audio stimulus not part of the original piece of music. Using the ERP paradigm described above, salient audio features or foreign auditory probes can improve the accuracy of estimating the user's current brain state because of the novelty of the audio will tend to increase the size of the user's ERP. For instance, a user's level of engagement can be estimated using a stimulus response paradigm where the stimulus is unexpected.

Meditation Example

The system architecture may implement the following work flow. Step 1: Set up a user's posture using audio feedback. Guide a person into postural alignment using sound.

A user's posture may be set up using accelerometer and sound to give them feedback. A user may be instructed to sit in an upright posture. Once a user's posture starts to stabilize a tone would start “filling-up” and once they hit a crescendo because they have been in that position for a while. Then the user may be asked to move and shift another part of their body. This may cause/destabilize the accelerometer threshold to reset and the crescendo may change alerting the user that their posture has changed. This process teaches the user how the posture correction system works. This creates a 3 dimensional feeling, they feel held, and when one is aided to adjust their body in a unique way they feel that the system is responsive. The users not only feel that the system is reactive but they are being “held” in that posture. The user's perception of the system was acquired using a phenomenological method of inquiry. The approach was built bottom-up using the legs, moving side-to-side, lifting the chest up, dropping the shoulders down, moving the head one way to the next, (left-right, tilt up-down). Proper posture gives more energy to the nervous system, if the body is alert then the mind will be more alert as well. Users have given the following kinds of feedback: “Wow, I never realized how important my body is for an alert mind.” The posture setting has a different paradigm than the sound paradigm given for EEG biofeedback (neuro-feedback) described herein.

Future enhancements of the posture algorithm can be used to show the user their posture across the session. This may reveal patterns of behaviour that can help the user understand and improve their posture performance.

Step 2: Instantaneous feedback of a person's brain state. There are several time scales of feedback provided to the user. The first and shortest time scale is moment to moment feedback which is instantaneous feedback. A synthesized tone is generated based on the analysis and interpretation of EEG that changes rapidly from one moment to the next. Example of brain state. This feedback is driven by a BUSY-MIND score which varies between 0 and 1. With 0 being a calm mind state and 1 being the busiest of mind states. See for example Applicant's U.S. application Ser. No. 14/368,333 entitled ADAPTIVE BRAIN TRAINING COMPUTER SYSTEM AND METHOD for a description of how a busy-mind score is calculated, the entirety of which is incorporated by reference herein.

Step 3. Maintain a target state. The next level is a trait feedback when one has maintained a certain state for a minimum period of time or a cumulative feedback. Once the person's busy mind score stays below a threshold (lower the score, the calmer the mind) a tone starts building up rising to a crescendo and then if one maintains the crescendo then it “unravels” into an interesting musical story. Once a person goes above the threshold then the crescendo stops building and is reset to the beginning. From a human interaction perspective this experience there are two points. The sound feedback is layered. Maintaining a target state is less judgemental and more welcoming than instantaneous biofeedback method. It opens up our app to allow it to be possibly used for other exercises such as body scan that require longer periods of feedback. The system is more generous with the feedback that it provides the user. This method is for people who have experience with the current instantaneous method of feedback. People with experience will understand the longer term state that they are being given feedback towards. It may be suitable for people without neuro-feedback experience but a way of showing them how the system responds to their mental state needs to be determined. This type of feedback encourages sustained, diligent practice.

Future improvements of this technology may include providing different sound paradigms for each zone of the BUSY-MIND score. The first threshold is set at 0.7 and this triggers the first sound paradigm. The next threshold of the BUSY-MIND score is set at 0.3. Sustaining the score below 0.3 triggers the next sound paradigm.

DJ System for Meditation Teacher Example

The meditation teacher is able to hear the brain state of each user in a live guided meditation class. A class comprises of up to half a dozen users sitting with headphones and brain sensing headbands. Each user in the audience is having their brain-state being fed back to them. In addition, the feedback of each user is fed to a mixer board that the instructor is listening. The instructor can toggle between each user to hear the brain state of each user. The instructor can also see the brain-state of each user in the audience. The instructor can see each user, the visual brain-state score and hear how the sound is modulated for that user. For instance, a user may be falling asleep and the instructor adjusts their control panel so that user is jarred awake using a sound designed to wake them. In another example of instructor-audience interaction, the instructor would notice that a user is holding a good meditative state. The instructor may decide to further challenge that user by increasing the difficulty threshold of the feedback for the second layer of sound feedback to maintain a target state. In a third example, a user may be having difficulty maintaining a meditative state, in which case the thresholds may be relaxed by the instructor to help user get through the practice.

The thresholds of the feedback may be changing and being manipulated by the instructor. The instructor may be telling the users that he/she will be controlling certain aspects of what the user is going to hear. The users are told that if the instructor sees that a user has a stable breathing pattern then the instructor will change the type of feedback given to the user. The instructor can also choose to bring the audience to a shared experience through musical language. The audience can be cued to create a certain brain state. All of the audience can be brought into a meditative state. The audience can also be agitated using higher frequencies or put into a state of relaxation using lower frequency tones. The audio feedback given to the users need not be biofeedback but can be created by the instructor. The instructor may choose for instance to play a sound track that is similar to the collective mood that the users are in. The instructor develops intuition as to where the audience is at and what they need. This system may also help new facilitators learn how to interpret an audience because they are given additional information about the physiological state of each member of the audience. Other measurements can include group measures of the audience. Scores that the audience together contributes to. Another aspect of this disclosure is that the instructor can bring the audience into electrophysiological synchrony of their heart beats, breathing rate, and EEG patterns. Physiological synchrony can be enhanced using EEG. There could be a hierarchy that starts with synchronized breathing, then synchronized heart rate, muscle tension, and synchronized EEG state.

This platform can be used to get labelled data about each user in the audience. The instructor can interpret the state of the user based on the user's posture, facial expression, breathing rate, and EEG scores. The experienced instructor can label the data relevant to the meditation experience as user 1 is in state 2, user 2 is in state 4, etc. This can be used as a way to codify the instructor and automate the process of instruction that can be transformed into an application using machine learning techniques.

Networked Meditation Example

A user is doing a meditation session guided by scores driven from analysis of EEG picked up through brain-sensing headband. Through special tones a user is alerted to another user who is meditating in another location, possibly other side of the world. Each user can hear significant events and the state of another person's brain state while each person meditates. This provides a felt sense of presence for each participant sharing their meditative state and hearing the state of others' meditation session. Initially the appearance of another meditator could be signalled by a unique piece of music, i.e. musical appearance. Other states of that meditator can be represented using similar musical theme as their musical appearance. The state of the other meditator can also be music modulated by their state of meditation. After the session is over, the application can alert the user that another person was meditating at the same time and the link to the other person's profile is shown. The other meditator could be somebody on your friend list.

Emotion Scoring Engine Example

For recommendations, the engine processor may use an emotion scoring example. This may be referred to as an Emotion Scoring Engine.

Emotion Scoring Engine may be configured to detect and score the following examples.

Example emotions: Unconscious positive or negative reactions.

Affect: When one becomes conscious of their emotions. In other words feelings.

Mood: A general state based on the average of emotions felt throughout the day.

There may be universal time stamping of music tracks. Music tracks are standardized as to when a track starts so that time-stamps are universal across all presentations of a specific track of music.

With digital technology, tracks may be clipped. It is important when tagging a track with emotion that it corresponds with the music event (e.g. note) that evoked that emotion.

The Emotion Scoring Engine may have EEG Scoring. This may be labelled EEG data as to the type of music a person was listening to is collected and stored in Cloud data. Machine learning is used to develop a classifier of the EEG. EEG can also be clustered based on individual and the type of music that a person was listening to.

The Emotion Scoring Engine may connect to sensors such as Accelerometer and Gyroscope.

This may classify rhythm in music and match to user's movement, or detect any rhythm in user's movement. If a user is moving with rhythm then this can be used as an input to classify the level of engagement of a person with the user.

For ECG, the heart rate may help classify a person's emotion.

Example Application: A Journey Use Story

The following is an illustrative shared user experience user story for this example application.

A user is in a wellness studio, a quiet calming environment, with individual meditation spots with a comfortable mat and meditation cushions. The user makes himself comfortable, slips a head-worn device, containing bio-signal sensors, around his head, and places a light and sound orb where the user desires. The orb lights up and emits a gentle sound as the bio-signal sensors begin to sense the user's brain and heart. At the front of the room there are guides for the day's session, a bamboo flute artist and a meditation teacher who holds a grandmother drum engage with people as other users flow into the room and get settled.

Everyone has taken their seat and the doors close. Lights dim until all that's left is the glow of the orbs.

A chime emanates from the front. The teacher has started the session. The user hears gentle sonorous drone start to emanate from their orb. Following her guidance, the user shifts to adjust their posture, takes a deep breath and their orb responds with a familiar warble in the drone and a patter of chimes. The user closes their eyes, and the sound starts to change, like the petals of a flower opening up, following the changes in the user's brainwaves as the user lets go and allow sensations to flow through the user's consciousness.

The user opens up their awareness to the sound in the room, hearing other users go through a similar process as he prepares for the journey ahead. The room feels alive. A soft harmony of minds hangs in the air. The outside world is gone.

Out of the calm anticipation, the flute begins to speak. It's breathy sound, much like the human voice. Clear and nuanced, the user can feel the texture of the air as it brings the whole room into its resonance.

The user begins to hear the sound of the drum, quietly drumming out a relaxed heartbeat rhythm adding to the improvisation of the flute. With the instructor's voice and the flute and drum guiding the user, the user begins to breath with the rhythm. As the user's heart starts to respond, the user's orb sings out with the variations in the user's heartbeat, high and low vocal tones that weave a beautiful harmony with the flute, and a soft wave of bass that pulses with the beat of the user's heart. A flood of colour pours through the user's eyelids, and the user opens them to see a beautiful moving colourfield, shown by way of example in FIG. 20A, flowing with rhythm of the breath.

All around the user can be heard the chorus of hearts, singing in key with the flute, Some high, some low, mirroring the uniqueness of each person in the group.

The cadences of the breath rises, energizing the room. The bass undertones create an earthy wash that feels like it's supporting the user's body.

The cadence slows. The user notices that with deep settling exhales that he is sinking deeper into meditation. There is only the wave of the breath and a flow of vibration. The user notices that the wash of bass in the room, has become a harmonic throb, that is moving with the drum. The hearts of the users in the room are aligned. when did that happen? And in that moment of realization, the colours start to change, shown by way of example in FIG. 20B. The users are moving into a focus practice.

Eyes half lidded, with the orb in front of him, the user stabilizes their gaze within its depths. The user's focus transforming the colour of the orb, shown by way of example in FIG. 20C. On the reflective surface of the orb see geometric patterns evolve, shown for example in FIG. 20D. The patterns are project onto the ceiling of the studio, reflecting the focus of the group. The patterns seem like the are from another world, and the orb as the portal into that universe.

The more the user stabilizes his focus, the more resonant and harmonious is the choir-like sound that comes from the orb. As the group strives to find the one pointed concentration of deep meditation, the voices come together in a crystal like harmony, feather light and pure, like a starry night, visualized for example in FIG. 20E.

As night turns to dawn, the user hears birds and the whisper of wind through leaves, and knows that the end of the journey is near, arriving back on earth with the familiar sounds of nature, and visualized for example in FIG. 20F. The user knows through the group experience that each person contributes to the flow of the wind, the tinkle of chimes and the flow of resonant harmonies that hang in the air.

The guide leads the users to connect with feeling of connection between themselves, and the source of inspiration that they all have within them. The flutist begins to play again and together the users let the improvisation happen. The play of the users' minds, the wind in the trees, the flow of the users breath, bodies and hearts. Wedding the users' inner symphony with the delicate touch that the last 45 minutes allows them to access.

The user hears birds flowing in from far away, piped in through speakers that connect to a network. A whole community shared in this journey, courtesy of a binaural microphone in the middle of the room. Having also joined in from home, others could hear everything that was happening as if they were present in the room.

General

It will be appreciated that any module or component exemplified herein that executes instructions may include or otherwise have access to computer readable media such as storage media, computer storage media, or data storage devices (removable and/or non-removable) such as, for example, magnetic disks, optical disks, tape, and other forms of computer readable media. Computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. Examples of computer storage media include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD), blue-ray disks, or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by an application, module, or both. Any such computer storage media may be part of the mobile device, tracking module, object tracking application, etc., or accessible or connectable thereto. Any application or module herein described may be implemented using computer readable/executable instructions that may be stored or otherwise held by such computer readable media.

Thus, alterations, modifications and variations can be effected to the particular embodiments by those of skill in the art without departing from the scope of this disclosure, which is defined solely by the claims appended hereto.

The present systems and methods may be practiced in various embodiments. A suitably configured computer device, and associated communications networks, devices, software and firmware may provide a platform for enabling one or more embodiments as described above, by way of example, computing device 500 as shown in FIG. 9.

In further aspects, the disclosure provides systems, devices, methods, and computer programming products, including non-transient machine-readable instruction sets, for use in implementing such methods and enabling the functionality described previously.

Although the disclosure has been described and illustrated in exemplary forms with a certain degree of particularity, it is noted that the description and illustrations have been made by way of example only. Numerous changes in the details of construction and combination and arrangement of parts and steps may be made. Accordingly, such changes are intended to be included in the disclosure, the scope of which is defined by the claims.

Except to the extent explicitly stated or inherent within the processes described, including any optional steps or components thereof, no required order, sequence, or combination is intended or implied. As will be will be understood by those skilled in the relevant arts, with respect to both processes and any systems, devices, etc., described herein, a wide range of variations is possible, and even advantageous, in various circumstances, without departing from the scope of the invention, which is to be limited only by the claims. 

What is claimed is:
 1. A computer-implemented method for sharing a user experience, comprising: receiving bio-signal data for a plurality of users from a plurality of bio-signal sensors; receiving sound data representing sound experienced by each of the users; determining a physiological state of each of the users based at least in part on the bio-signal data of that user; determining an environmental state associated with each of the users based at least in part on the sound data; using a rules engine, determining if a condition is met, based at least in part on the physiological state of each of the users and the environmental state associated with each user; upon the condition being met, executing an associated action including generating a sensory signal; and sending the sensory signal to transducers to output a sensory output to each of the users based on the sensory signal.
 2. The method of claim 1, further comprising: detecting an EEG response based on the bio-signal data; and correlating a time of the EEG response to the sound data to compute a segment of sound data corresponding to the EEG response, wherein the sensory signal comprises an audio signal, the audio signal based at least in part on the EEG response and the corresponding segment of sound data.
 3. The method of claim 1, wherein the sound data is received from an acoustic sensor proximate one or more of the users.
 4. The method of claim 1, wherein the bio-signal data includes at least one of brain metrics, heart metrics, or breath metrics.
 5. The method of claim 1, wherein the sensory output includes at least one of light, sound, tactile pressure, tactile vibration, or scent.
 6. The method of claim 1, further comprising: receiving image data from one or more image sensors proximate one or more of the users, wherein the determining the environmental state associated with each of the users is based at least in part on the image data.
 7. The method of claim 1, further comprising: receiving image data from one or more image sensors proximate one or more of the users, wherein the physiological state associated with each of the users is based at least in part on the image data.
 8. The method of claim 1, further comprising: receiving tactile signals from one or more tactile transducers proximate one or more of the users, wherein the determining the physiological state associated with each of the users is based at least in part on the tactile signals.
 9. The method of claim 1, wherein the sensory output to each of the users is individualized to each user based at least in part on a user profile of that user.
 10. A non-transitory computer-readable medium having computer executable instructions stored thereon for execution by one or more computing devices, that when executed perform the method of claim
 1. 11. A shared user experience system, comprising: a plurality of bio-signal sensors to capture bio-signal data from a plurality of users; an input device for at least one user to change a state of a rules engine; a plurality of transducers to output a sensory output to each of the users; and a computing device in communication with the bio-signal sensors, the acoustic sensor, and the transducers, the computing device configured to: receive bio-signal data for the users from the bio-signal sensors; receive input data from the input device; determine a physiological state of each of the users based at least in part on the bio-signal data of that user; using the rules engine, determine if a condition is met, based at least in part on the physiological state of each of the users; and upon the condition being met, execute an associated action including generate a sensory signal and send the sensory signal to the transducers to output a sensory output to the users based on the sensory signal.
 12. The system of claim 11, wherein the bio-signal data includes at least one of brain metrics, heart metrics, or breath metrics.
 13. The system of claim 11, wherein the sensory output includes one or more of visual, audio, tactile, heat, or scent.
 14. The system of claim 11, further comprising a tactile transducer to receive tactile input from the users, wherein the computing device is in communication with the tactile transducer and is further configured to: receive tactile signals from the tactile transducer, and the physiological state of each of the users is based at least in part on the tactile signals of that user.
 15. The system of claim 11, wherein the computing device is further configured to: determine a group physiological state of the users based at least in part on the bio-signal data of each user and the state of the rules engine, wherein the condition is based at least in part on the group physiological state and the state of the rules engine.
 16. The system of claim 11, wherein the computing device is further configured to: receive sound data from one or more acoustic sensors and determine an environmental state associated with each of the users based at least in part on the sound data, wherein the condition is based at least in part on the physiological state of each of the users and the environmental state associated with each user.
 17. The system of claim 16, wherein the physiological state of each of the users is based at least in part on the bio-signal data of that user and time-stamped sound data as measured by the one or more acoustic sensors representing sound delivered to each of the users, the bio-signal data time-stamped and synchronized with the time-stamped sound data.
 18. The system of claim 11, wherein one or more of the users are located remotely from each other.
 19. The system of claim 11, wherein the computing device is in communication with the bio-signal sensors, the acoustic sensor, and the transducers over a network.
 20. The system of claim 11, wherein the condition is based at least in part on a performance of the physiological states of the users as compared to a target physiological state. 